nextflow-io / nextflow-io/nextflow

Globstar output patterns may result in invalid move operations when `scratch = true` and `stageOutMode = move`

Open
#7,637 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage/investigate
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

Expected behavior and actual behavior

Expected: a process that declares a ** output alongside other outputs under the same root should unstage all of them successfully, regardless of scratch and stageOutMode.

Actual: when scratch = true and stageOutMode = move, the process may fail to unstage some outputs correctly because their parents have already been moved.

normalizeGlobStarPaths() truncates every ** pattern to its longest wildcard-free parent (e.g., out/dir/**/*.txt becomes out/dir).

When that parent is also an ancestor of another declared output, this will result in an invalid move operation sequence, as either the parent is moved before the child, leaving the child path nonexistent, or the child is moved first, potentially causing conflicts with the parent move.

Steps to reproduce the problem

Many nf-core pipelines currently fail when using scratch = true and stageOutMode = move. I have encountered this bug in the most recent versions of nf-core/demultiplex and nf-core/rnafusion, but I strongly suspect there are more.

Here's a minimal reproducer:

// repro.nf

process REPRO {
    scratch true
    stageOutMode 'move'

    output:
    path("out/**/*.txt")
    path("out/meta")

    script:
    """
    mkdir -p out/sub out/meta
    touch out/sub/a.txt out/meta/b
    """
}

workflow {
    REPRO()
}
Program output
$ nextflow run repro.nf

 N E X T F L O W   ~  version 26.04.6

Launching `repro.nf` [suspicious_leibniz] revision: 90cb1e2e3c

executor >  local (1)
[6c/de61c5] process > REPRO [  0%] 0 of 1 ✘
ERROR ~ Error executing process > 'REPRO'

Caused by:
  Process `REPRO` terminated with an error exit status (1)


Command executed:

  mkdir -p out/sub out/meta
  touch out/sub/a.txt out/meta/b

Command exit status:
  1

Command output:
  (empty)

Command error:
  mv: cannot stat 'out/meta': No such file or directory

Work dir:
  /Users/erik/nxf/work/6c/de61c521d0a4ab21bc7baaedce36c6

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

Looking at the generated .command.run, here's the problelmatic part:

nxf_unstage_outputs() {
    true
    IFS=$'\n'
    for name in $(eval "ls -1d out out/meta" | sort | uniq); do
        nxf_fs_move "$name" /Users/erik/nxf/work/6c/de61c521d0a4ab21bc7baaedce36c6
    done
    unset IFS
}

Notice how out is listed before out/meta in the ls command, which causes the parent directory to be moved before its subdirectory, leading to the error.

Environment
  • Nextflow version: 26.04.6 build 12646
  • Java version: Groovy 4.0.31 on OpenJDK 64-Bit Server VM 25.0.2+10-LTS
  • Operating system: Mac OS X 26.6.2
  • Bash version: zsh 5.9 (arm64-apple-darwin25.0)
Additional context

A potential fix for this issue is to check if a failed move operation is due to a parent already having been moved, and if so, skip the move for that output.

nxf_fs_move() {
  local err
  local source=$1
  local target=$2
  local basedir=$(dirname "$1")

  mkdir -p "$target/$basedir"
  err=$(mv -f "$source" "$target/$basedir" 2>&1) && return 0

  if { [ ! -e "$source" ] && [ ! -L "$source" ]; } \
  && { [ -e "$target/$source" ] || [ -L "$target/$source" ]; }; then
      echo "Target $target/$source already moved, skipping"
      return 0
  else
      echo "Failed to unstage output: $source -- ${err}" >&2
      return 1
  fi
}

I'm not sure if there are any edge cases that this approach might not cover, but from my limited testing it seems to handle the common scenarios correctly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the provided repro.nf, then trace normalizeGlobStarPaths() and the generated nxf_unstage_outputs()/nxf_fs_move() entry points. Check how parent and child output paths are ordered during move operations. Done means all declared outputs unstage successfully with scratch true and stageOutMode 'move', including the reproducer.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, groovy
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.