Loop in Nextflow logic caused thousands of empty folders in S3 staging folder inside workdir

Bug report

Unclear on exactly how this happened, but our code set some paths some where explicitly to “/“ or an empty string. This worked locally, however, when that code ran in AWS Batch with a S3 workdir it caused an endless loop. The error we got was:

WARN: Path integrity check failed because the following file has been deleted: s3://bucket-name/pipeline-workdirs/job-id/stage-b0b335df-e31b-4612-a89e-9d2d3d56e41f/2c/fa25ce3231be63235902b85b01769e/bin -- make sure to not run more than one nextflow instance using the same work directory

Within the root of the workdir on S3 was the folder: stage-b0b335df-e31b-4612-a89e-9d2d3d56e41f

Within that stage folder was many thousand empty folders with a 2 character path followed by random string path like: /2c/fa25ce3231be63235902b85b01769e

None of the empty folders had a subfolder of bin (mentioned in the error) or any files in them.

Expected behavior and actual behavior

Expected - If a file is deleted or removed in S3 (or in our case just doesn't exist), then cancel the loop, throw error and end job.

Actual - Error detected that file in S3 might have been deleted. Job kept retrying over and over to stage something, which resulted in an empty folder each time.

Steps to reproduce the problem

I'm not sure how to reproduce without providing our code which I cannot do in its current form.

Program output

The error looks to be from here: https://github.com/nextflow-io/nextflow/blob/d67aef5fecf09132a02276cc6735d431b7c84c42/modules/nextflow/src/main/groovy/nextflow/file/FilePorter.groovy#L405

And the loop calling it here: https://github.com/nextflow-io/nextflow/blob/d67aef5fecf09132a02276cc6735d431b7c84c42/modules/nextflow/src/main/groovy/nextflow/file/FilePorter.groovy#L375

Environment

• Nextflow version: 23.04.4
• Java version: 17
• Operating system: Linux
• Bash version: Unknown

Additional context

Within 10 minutes this loop caused over 1 million log entries and many thousand empty folders within a stage directory inside the workdir on S3. With this many S3 and CloudWatch API calls, it could become an expensive problem if it had been left running.

In our setup, we do not get any .nextflow.log files from a failed Batch job when manually terminated. So no further info to provide.

The workdir location in S3 was a new work directory, despite the error suggesting it may not be.

I guess the question I'm asking is, should Nextflow continue to try staging files to S3 in a scenario where it can’t find the file, or should the loop exit and the job error out?

Or is it just a client side issue as our code was poorly written? If so, then feel free to close this issue but it feels worth raising this due to the potential of an endless loop when working with S3.

Start with the loop around FilePorter.groovy lines 375 and 405, then review the S3 staging behavior described in the report. Investigate how a missing file or failed path-integrity check is retried and verify the behavior with the reported AWS Batch and S3 workdir scenario. Done means the failure no longer creates unbounded empty folders or log entries.