nextflow-io / nextflow-io/nextflow
Inputs included as outputs are needlessly unstaged from the scratch directory
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Bug report
Many users will use the scratch true directive, in part to minimize the size of the shared work directory - to ensure that the files saved to the work directory are restricted to only those necessary for downstream tasks and for the resume mechanism.
In cases where a process outputs glob pattern also matches the input file, the input file is unnecessarily copied back into the shared work directory
Steps to reproduce the problem
Given main.nf:
process GreedyOutputGlob {
scratch true
input: path(csv)
output: path("*.csv")
script: "cp $csv out.csv"
}
workflow {
Channel.fromPath("data/in.csv")
| GreedyOutputGlob
| view
}
Note that the in.csv file is copied back to the shared work directory:
❯ nextflow run .
N E X T F L O W ~ version 23.04.1
Launching `./main.nf` [hopeful_church] DSL2 - revision: 06d2458686
executor > local (1)
[42/2fa08b] process > GreedyOutputGlob (1) [100%] 1 of 1 ✔
/private/tmp/foo/work/42/2fa08b2ef83cd1799c58833592deed/out.csv
/tmp/foo on ☁️ sts on ☁️ devstar2002@gcplab.me took 2s
❯ tree work
work
└── 42
└── 2fa08b2ef83cd1799c58833592deed
├── in.csv
└── out.csv
3 directories, 2 files
This is because the nxf_unstage command uses the output glob pattern directly, without regard to the input files:
# ...
for name in $(eval "ls -1d *.csv" | sort | uniq); do
nxf_fs_copy "$name" /private/tmp/foo/work/42/2fa08b2ef83cd1799c58833592deed || true
done
# ...
Expected behaviour and actual behaviour
To help users save storing the duplicated input files, it would be better if Nextflow excluded input files from being copied back to the shared work directory (unless the includeInputs: true argument is included in the outputs: block).
Environment
- Nextflow version: 23.04.1
- Java version: openjdk version "17.0.5" 2022-10-18
- Operating system: all
- Bash version: all
(Add any other context about the problem here)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the main.nf reproduction and trace the nxf_unstage command that handles the output glob. Check how input files and the outputs includeInputs option are represented, then verify that the input is not copied back to the shared work directory unless explicitly requested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100