Question 47039 in Debian Question posted in
The oficial documentation can be found here.

Spring Cloud Dataflow Duplicate key error (A job execution for this job is already running) – Debian

I have a Spring Cloud Dataflow Stream with an Http source, custom processor and Task-Launcher
enter image description here

I have a composed task that gets called from the stream – task-launcher sink

enter image description here

The composed task gets launched correctly and runs successfully when only 1 instance is executed but when multiple instances are created they intermittently have errors with duplicate keys. I have recently updated to the following versions:

• bitnami/spring-cloud-dataflow 
version: 2.9.0-debian-10-r15
• bitnami/spring-cloud-dataflow-composed-task-runner
version: 2.9.0-debian-10-r17
• bitnami/spring-cloud-skipper
version: 2.8.0-debian-10-r15

java.lang.IllegalStateException: Failed to execute ApplicationRunner
2021-10-28T16:53:59.179909948Z  at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:785) [spring-boot-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.179913386Z  at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:772) [spring-boot-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.179916469Z  at org.springframework.boot.SpringApplication.run(SpringApplication.java:345) [spring-boot-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.179919669Z  at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) [spring-boot-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.179922662Z  at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332) [spring-boot-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.179925769Z  at org.springframework.cloud.dataflow.composedtaskrunner.ComposedTaskRunner.main(ComposedTaskRunner.java:31) [classes!/:2.9.0]
2021-10-28T16:53:59.179928817Z  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_312]
2021-10-28T16:53:59.179931884Z  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_312]
2021-10-28T16:53:59.179934872Z  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312]
2021-10-28T16:53:59.179937847Z  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312]
2021-10-28T16:53:59.179940922Z  at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [spring-cloud-dataflow-composed-task-runner-2.9.0.jar:2.9.0]
2021-10-28T16:53:59.179944118Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) [spring-cloud-dataflow-composed-task-runner-2.9.0.jar:2.9.0]
2021-10-28T16:53:59.179947158Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [spring-cloud-dataflow-composed-task-runner-2.9.0.jar:2.9.0]
2021-10-28T16:53:59.179950209Z  at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) [spring-cloud-dataflow-composed-task-runner-2.9.0.jar:2.9.0]
2021-10-28T16:53:59.179953277Z Caused by: org.springframework.batch.core.repository.JobExecutionAlreadyRunningException: A job execution for this job is already running: JobInstance: id=1200, version=0, Job=[composedtask]
2021-10-28T16:53:59.179957212Z  at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:127) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
2021-10-28T16:53:59.179960544Z  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_312]
2021-10-28T16:53:59.179963557Z  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_312]
2021-10-28T16:53:59.179975310Z  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312]
2021-10-28T16:53:59.179978588Z  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312]
2021-10-28T16:53:59.179981557Z  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.179984657Z  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.179987854Z  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.179990997Z  at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.179994173Z  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.179997313Z  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180000409Z  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180003545Z  at org.springframework.batch.core.repository.support.AbstractJobRepositoryFactoryBean$1.invoke(AbstractJobRepositoryFactoryBean.java:181) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
2021-10-28T16:53:59.180006834Z  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180009937Z  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180013021Z  at com.sun.proxy.$Proxy64.createJobExecution(Unknown Source) ~[na:na]
2021-10-28T16:53:59.180016256Z  at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:137) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
2021-10-28T16:53:59.180019375Z  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_312]
2021-10-28T16:53:59.180022367Z  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_312]
2021-10-28T16:53:59.180025349Z  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312]
2021-10-28T16:53:59.180028473Z  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312]
2021-10-28T16:53:59.180031405Z  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180034563Z  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180037663Z  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180042900Z  at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
2021-10-28T16:53:59.180046230Z  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180049362Z  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180052538Z  at com.sun.proxy.$Proxy67.run(Unknown Source) ~[na:na]
2021-10-28T16:53:59.180055536Z  at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.execute(JobLauncherApplicationRunner.java:199) ~[spring-boot-autoconfigure-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.180058822Z  at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.executeLocalJobs(JobLauncherApplicationRunner.java:173) ~[spring-boot-autoconfigure-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.180061980Z  at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.launchJobFromProperties(JobLauncherApplicationRunner.java:160) ~[spring-boot-autoconfigure-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.180065453Z  at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:155) ~[spring-boot-autoconfigure-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.180068728Z  at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:150) ~[spring-boot-autoconfigure-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.180071912Z  at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:782) [spring-boot-2.5.5.jar!/:2.5.5]

I reported this issue a few months back when we were running scdf 2.8.1, I was told that the issue had been acknowledged and would be corrected in a future release. Any updates on this issue or workarounds?

I have tried using the –increment-instance-enabled=true on the task-launcher sink but this did not help.

Thanks!

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    After upgrading Spring Cloud Dataflow to 2.9.0

    I Finally figured this out, with some help from VMWare. The answer is to set the uuid-instance-enabled property=true.

    Initially I was trying to set this on the stream deployment properties for the task-launcher and this was incorrect.

    In the custom processor where the task-launch request message is created, I set the uuid-instance-enabled=true property and this corrected the issue. The properties set here will filter down to the composed task runner that is spun up (kubernetes pod) to execute each composed task.

    Example code:

    List<String> composedTaskArgs = new ArrayList<>();
composedTaskArgs.add("--dataflow-server-uri="+scdfProperties.getDataflowServerUri());
composedTaskArgs.add("--uuid-instance-enabled=true");

json.put("args", composedTaskArgs);
json.put("deploymentProps", props);

message = json.toString();

logger.info("Send to tasklauncher: "+message);
return new GenericMessage<>(message);

  2. 0

    Unfortunately, it appears, we didn’t backport the bug-fix (ie: 4684) to the 2.8.x branch, so that’s why it is not yet available.

    That said, though, this fix is included in the latest 2.9.0/2.9.1 releases — please give it a go and let us know if you see an anomaly.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search