Race condition in stage-in of ./bin/ on AWS Batch running ECS Docker containers

Bug report

Hi there,

We're running Nextflow workflows on AWS Batch. For workflows with 1000s of similar jobs and >500 running at a time, rarely, we get this .command.log consisting of just a single line, and Nextflow reports exit code 255 for the process:

[Errno 17] File exists: '/tmp/nextflow-bin'

I believe it is an aws s3 cp race condition with the stage-in of the ./bin directory into the scratch = true space, i.e. /tmp.

This /tmp space is mounted across all Docker containers from an EBS /scratch space on the same EC2 host, from our nextflow.config:

volumes = ['/scratch:/tmp']

Edit: Here seems to be the responsible code, so this is actually nf-amazon plugin code:
https://github.com/nextflow-io/nextflow/blob/1732562dc419080d6a87426888a0d6a840ea5a8f/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchFileCopyStrategy.groovy#L72

Expected behavior and actual behavior

• Expected: Multiple Nextflow processes running in independent Docker containers on the same EC2 host, and a shared /tmp space, do not overwrite each others files or otherwise conflict
• Actual: Rare /tmp/nextflow-bin conflicts that lead to early errored exits

Steps to reproduce the problem

(Rarely occurs)

Program output

From .nextflow.log the overall workflow only uploads bin to cloud storage once.

Jan-25 22:09:47.781 [main] INFO  n.cloud.aws.batch.AwsBatchExecutor - Uploading local `bin` scripts folder to s3://vfvf-nextflow/logs/ngs-pipeline-orchestrator/sample-2026-01-25-22-00-02/tmp/e0/a4571e0347e77b7829c8bffc597a5c/bin

In the current workflow I am troubleshooting, there are 3 errored processes that were not retried.

Copying a snippet from nxf_main from the .command.run of those errored processes:

    [[ "${NXF_CHDIR:-}" ]] && cd "$NXF_CHDIR"
    NXF_SCRATCH="$(set +u; nxf_mktemp $TMPDIR)"
    [[ $NXF_DEBUG > 0 ]] && nxf_env
    echo start | nxf_s3_upload - s3://vfvf-nextflow/logs/ngs-pipeline-orchestrator/sample-2026-01-25-22-00-02/a9/64a0b72d1f74e6a530b6f9d2baa9ae/.command.begin
    set +u
    set -u
    /home/ec2-user/miniconda/bin/aws --region us-east-1 s3 cp --recursive --only-show-errors s3://vfvf-nextflow/logs/ngs-pipeline-orchestrator/sample-2026-01-25-22-00-02/tmp/e0/a4571e0347e77b7829c8bffc597a5c/bin $PWD/nextflow-bin
    chmod +x $PWD/nextflow-bin/* || true
    export PATH=$PWD/nextflow-bin:$PATH

Note the $PWD/nextflow-bin above, that resolves to /tmp/nextflow-bin. Using, $NXF_SCRATCH/nextflow-bin would presumably avoid this conflict, i.e. something like /tmp/nxf.2PsaJ1tKep/nextflow-bin

Environment

• Nextflow version (on head node): 25.12.0-edge
• nf-amazon plugin version (on head node): 3.6.0
• Java version (on head node): OpenJDK Runtime Environment Corretto-21.0.9.10.1 (build 21.0.9+10-LTS)
• Java version (on EC2 host instance): OpenJDK Runtime Environment Corretto-17.0.6.10.1 (build 17.0.6+10-LTS)
• Operating system (on EC2 host instance): Amazon Linux 2
• Bash version (on EC2 host instance): GNU bash, version 4.2.46(2)-release (x86_64-koji-linux-gnu)
• Bash version (inside Docker container on EC2): GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)

Start in plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchFileCopyStrategy.groovy around line 72, where the stage-in command creates nextflow-bin. Reproduce concurrent AWS Batch processes using a shared /tmp mount, then verify that independent containers no longer collide when staging the bin directory and that the workflow avoids the exit-255 failure.