GoogleBatch errors about Project exists - How to handle retries

Bug report

Expected behavior and actual behavior

Hello everyone, every now and then we receive errors grom Google Batch API in our pipeline running on Google Cloud that we cannot handle by retries. These seem to be related to the Batch API and therefore nextflow fails to produce an exit code and the entire workflow fails.
For example:

ERROR ~ Error executing process > 'vcf_sort_zip_and_index_merged (1)'

Caused by:
  ALREADY_EXISTS: Resource "projects/907794457535/locations/europe-west1/jobs/nf-06071afa-1773255240303" already exists


 -- Check '/mnt/stateful_partition/work/logs/158294/20260311-1431/nextflow.log' file for details

Steps to reproduce the problem

The pipeline runs within a e2-medium machine that is launching batch jobs for tasks (processes that are intensive). The cmd used to launch nextflow can be found in the traceback below

subprocess.CalledProcessError: Command '['nextflow', '-log', '/mnt/stateful_partition/work/logs/158294/20260311-1431/nextflow.log', 'run', 'targeted_onco', '-pi', '60s', '-profile', 'googlecloud', '--task_id', '158294', '--git_branch', 'gcap_v8_helios', '--git_sha1_hash', '7bd6646246f0b8f8f7da585830e571cbb963d429', '--project_id', 'diagnostics-uz', '--topic_id', 'gcap_nf_service', '--pubsub_message_id', '18631455654071366']' returned non-zero exit status 1.

Program output

Mar-11 19:55:01.379 [Task submitter] ERROR nextflow.processor.TaskProcessor - Error executing process > 'vcf_sort_zip_and_index_merged (1)'

Caused by:
  ALREADY_EXISTS: Resource "projects/907794457535/locations/europe-west1/jobs/nf-06071afa-1773255240303" already exists


com.google.api.gax.rpc.AlreadyExistsException: io.grpc.StatusRuntimeException: ALREADY_EXISTS: Resource "projects/907794457535/locations/europe-west1/jobs/nf-06071afa-1773255240303" already exists
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:96)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1128)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)
        at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:568)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:538)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:570)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
        Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
                at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
                at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
                at com.google.cloud.batch.v1.BatchServiceClient.createJob(BatchServiceClient.java:316)
                at com.google.cloud.batch.v1.BatchServiceClient.createJob(BatchServiceClient.java:248)
                at nextflow.cloud.google.batch.client.BatchClient$_submitJob_lambda1.doCall(BatchClient.groovy:98)
                at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:236)
                at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
                at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:75)
                at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:176)
                at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:437)
                at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:115)
                at nextflow.cloud.google.batch.client.BatchClient.apply(BatchClient.groovy:187)
                at nextflow.cloud.google.batch.client.BatchClient.submitJob(BatchClient.groovy:98)
                at nextflow.cloud.google.batch.GoogleBatchTaskHandler.submit(GoogleBatchTaskHandler.groovy:166)
                at nextflow.processor.TaskPollingMonitor.submit(TaskPollingMonitor.groovy:218)
                at nextflow.processor.TaskPollingMonitor.submitPendingTasks(TaskPollingMonitor.groovy:599)
                at nextflow.processor.TaskPollingMonitor.submitLoop(TaskPollingMonitor.groovy:413)
                at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
                at java.base/java.lang.reflect.Method.invoke(Method.java:580)
                at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
                at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
                at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
                at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
                at groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
                at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
                at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
                at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
                at groovy.lang.Closure.call(Closure.java:433)
                at groovy.lang.Closure.call(Closure.java:412)
                at groovy.lang.Closure.run(Closure.java:505)
                at java.base/java.lang.VirtualThread.run(VirtualThread.java:309)
Caused by: io.grpc.StatusRuntimeException: ALREADY_EXISTS: Resource "projects/907794457535/locations/europe-west1/jobs/nf-06071afa-1773255240303" already exists
        at io.grpc.Status.asRuntimeException(Status.java:537)
        ... 14 common frames omitted
Mar-11 19:55:01.416 [Task submitter] DEBUG nextflow.Session - Session aborted -- Cause: io.grpc.StatusRuntimeException: ALREADY_EXISTS: Resource "projects/907794457535/locations/europe-west1/jobs/nf-06071afa-1773255240303" already exists

Environment

• Nextflow version: 24.04.4
• Operating system: Linux
• Java: openjdk 21.0.1 2023-10-17
• Bash version: GNU bash, version 5.2.21(1)-release (x86_64-pc-linux-gnu)

Additional context

The Turn-around time for our pipeline is short since it is developed for diagnostic purposes and we would like to minimize such errors by implementing exception or retry strategies to avoid restarting (which has to be done from the beginning -> we do not use the -resume option due to our internal execution logging). Below is the configuration from our config

googlecloud {
    bucketDir = "gs://gcap-nf-scratch"
    docker.enabled = true

    process {
      withLabel: lotsOfWork {
        executor = 'google-batch'
        errorStrategy = { sleep(Math.pow(2, task.attempt) * 200 as long); return 'retry' }
        maxRetries = 10
      }

      // Default configuration for processes without labels
      errorStrategy = { sleep(Math.pow(2, task.attempt) * 200 as long); return 'retry' }
      maxRetries = 10
    }

    google {
      batch.spot = false
      batch.requestTimeout = '120 sec'
      location = 'europe-west1'
      project = 'diagnostics-uz'
      httpReadTimeout = '1h'
      httpConnectTimeout = '1h'
      storage.maxDelay = '1h'
      client.deadline = '120 sec'
      retries = 5
      retryInterval = '20 sec'
      maxRetryInterval = '2 min'
      batch.debug = true
      batch.sshDaemon = true
      batch.bootDiskSize = '100.GB'
    }

 }
}

Start with the Google Batch submission path shown in nextflow.cloud.google.batch.client.BatchClient.groovy, especially submitJob and apply, then trace how GoogleBatchTaskHandler.groovy handles the failed submission. Reproduce the ALREADY_EXISTS response with the reported retry configuration and verify that the resulting behavior no longer aborts the workflow unexpectedly.