Different Singularity behaviours between 22.10.7 vs 23.04.0+

Bug report

When users try to run this pipeline with Singularity, it works on Nextflow 22.10.7 and before, but fails on Nextflow 23.04.0 and later (including the latest release).

The failure happens when srst2 within the container try to run Bowtie2, where Bowtie2 attempts to create FIFO files under /tmp via mkfifo. As Singularity mounts /tmp by default, when multiple processes is running srst2, all of them will write their FIFO files to the host /tmp directory.

In Nextflow 22.10.7 and before, the FIFO files have longer file names, e.g. 124813.inpipe1, 124813.inpipe2, 124874.inpipe1, 124874.inpipe2, 124964.inpipe1, 124964.inpipe2, and all is well.

However, in Nextflow 23.04.0 and later, the FIFO files have much shorter file names, e.g. 61.inpipe1, 61.inpipe2, 62.inpipe1, 62.inpipe2, 63.inpipe1, 63.inpipe2. And the relevant processes soon crash due to what I think is namespace conflict, and the error looks like this:

(ERR): mkfifo(/tmp/62.inpipe1) failed.

The only thing changed between my tests is the Nextflow executable version, nothing else. I compared the .command.run and .command.sh between runs, they all look identical (except the work dir paths).

At this point, I am wondering is there some hidden environment variables changed between these Nextflow versions that would affect the behaviours of Singularity?

This seems to be a related issue: https://github.com/nf-core/taxprofiler/issues/422

I am able to work around the issue by forcing each container to use a different subdirectory in /tmp by adding

singularity.runOptions = '-B $(mktemp -d):/tmp'

to nextflow.config

Expected behavior and actual behavior

• Expected: Singularity should always behave the same regardless of Nextflow version.

• Actual: Latest Nextflow introduce a new bug.

Steps to reproduce the problem

Run the pipeline with -profile sanger

Program output

N/A

Environment

• Nextflow version: 22.10.7, 23.04.0, 24.04.4
• Java version: OpenJDK 11.04.24, OpenJDK 17.0.12
• Operating system: Ubuntu 22.04, Ubuntu 22.04.5
• Bash version: GNU bash version 5.1.16(1)-release, zsh 5.8.1 (x86_64-ubuntu-linux-gnu)

Additional context

N/A

Reproduce the linked GBS-Typer-sanger-nf pipeline with -profile sanger under Nextflow 22.10.7 and 23.04.0 or later, using the reported Singularity setup. Compare the generated .command.run and .command.sh files and inspect the nextflow.config workaround; done means identifying the version-related change that causes conflicting FIFO paths or documenting a confirmed cause and fix.