nextflow-io / nextflow-io/nextflow
Globstar output patterns may result in invalid move operations when `scratch = true` and `stageOutMode = move`
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Bug report
Expected behavior and actual behavior
Expected: a process that declares a ** output alongside other outputs under the same root should unstage all of them successfully, regardless of scratch and stageOutMode.
Actual: when scratch = true and stageOutMode = move, the process may fail to unstage some outputs correctly because their parents have already been moved.
normalizeGlobStarPaths() truncates every ** pattern to its longest wildcard-free parent (e.g., out/dir/**/*.txt becomes out/dir).
When that parent is also an ancestor of another declared output, this will result in an invalid move operation sequence, as either the parent is moved before the child, leaving the child path nonexistent, or the child is moved first, potentially causing conflicts with the parent move.
Steps to reproduce the problem
Many nf-core pipelines currently fail when using scratch = true and stageOutMode = move. I have encountered this bug in the most recent versions of nf-core/demultiplex and nf-core/rnafusion, but I strongly suspect there are more.
Here's a minimal reproducer:
// repro.nf
process REPRO {
scratch true
stageOutMode 'move'
output:
path("out/**/*.txt")
path("out/meta")
script:
"""
mkdir -p out/sub out/meta
touch out/sub/a.txt out/meta/b
"""
}
workflow {
REPRO()
}
Program output
$ nextflow run repro.nf
N E X T F L O W ~ version 26.04.6
Launching `repro.nf` [suspicious_leibniz] revision: 90cb1e2e3c
executor > local (1)
[6c/de61c5] process > REPRO [ 0%] 0 of 1 ✘
ERROR ~ Error executing process > 'REPRO'
Caused by:
Process `REPRO` terminated with an error exit status (1)
Command executed:
mkdir -p out/sub out/meta
touch out/sub/a.txt out/meta/b
Command exit status:
1
Command output:
(empty)
Command error:
mv: cannot stat 'out/meta': No such file or directory
Work dir:
/Users/erik/nxf/work/6c/de61c521d0a4ab21bc7baaedce36c6
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
-- Check '.nextflow.log' file for details
Looking at the generated .command.run, here's the problelmatic part:
nxf_unstage_outputs() {
true
IFS=$'\n'
for name in $(eval "ls -1d out out/meta" | sort | uniq); do
nxf_fs_move "$name" /Users/erik/nxf/work/6c/de61c521d0a4ab21bc7baaedce36c6
done
unset IFS
}
Notice how out is listed before out/meta in the ls command, which causes the parent directory to be moved before its subdirectory, leading to the error.
Environment
- Nextflow version: 26.04.6 build 12646
- Java version: Groovy 4.0.31 on OpenJDK 64-Bit Server VM 25.0.2+10-LTS
- Operating system: Mac OS X 26.6.2
- Bash version: zsh 5.9 (arm64-apple-darwin25.0)
Additional context
A potential fix for this issue is to check if a failed move operation is due to a parent already having been moved, and if so, skip the move for that output.
nxf_fs_move() {
local err
local source=$1
local target=$2
local basedir=$(dirname "$1")
mkdir -p "$target/$basedir"
err=$(mv -f "$source" "$target/$basedir" 2>&1) && return 0
if { [ ! -e "$source" ] && [ ! -L "$source" ]; } \
&& { [ -e "$target/$source" ] || [ -L "$target/$source" ]; }; then
echo "Target $target/$source already moved, skipping"
return 0
else
echo "Failed to unstage output: $source -- ${err}" >&2
return 1
fi
}
I'm not sure if there are any edge cases that this approach might not cover, but from my limited testing it seems to handle the common scenarios correctly.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the provided repro.nf, then trace normalizeGlobStarPaths() and the generated nxf_unstage_outputs()/nxf_fs_move() entry points. Check how parent and child output paths are ordered during move operations. Done means all declared outputs unstage successfully with scratch true and stageOutMode 'move', including the reproducer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, groovy
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100