Template: Resume function can lead to inconsistencies in software version reporting
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 322
- Forks
- 255
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 5
Description
Description of the bug
The current nf-core template, software versions are collected without disabling the cache function of .collectFile():
// Collate and save software versions
softwareVersionsToYAML(ch_versions)
.collectFile(
storeDir: "${params.outdir}/pipeline_info",
name: 'nf_core_' + 'diseasemodulediscovery_software_' + 'mqc_' + 'versions.yml',
sort: true,
newLine: true
).set { ch_collated_versions }
This can lead to inconsistencies (missing/additional processes listed) in combination with multiple pipeline runs using the Nextflow resume function.
A fix would be to disable the collectFile caching:
// Collate and save software versions
softwareVersionsToYAML(ch_versions)
.collectFile(
storeDir: "${params.outdir}/pipeline_info",
name: 'nf_core_' + 'diseasemodulediscovery_software_' + 'mqc_' + 'versions.yml',
sort: true,
newLine: true,
cache: false
).set { ch_collated_versions }
Or to not provide a storeDir, but cache the file in the work directory instead.
Something similar can happen to all files created using collectFile, which are stored outside of the work directory with caching enabled.
Command used and terminal output
# run pipeline (first run)
nextflow run diseasemodulediscovery -r dev -profile test,docker --outdir results
# delete result directory
rm -r results
# run pipeline again with resume, but skip a step (second run)
nextflow run diseasemodulediscovery -r dev -profile test,docker --outdir results --skip_gprofiler -resume
# run full pipeline again with resume (third run)
nextflow run diseasemodulediscovery -r dev -profile test,docker --outdir results -resume
# the third run uses the caching of gprofiler from the first run (cached in work), but the caching of the software versions from the second run (cached in results)
# As a result, the software versions of gprofiler are not listed in the multiqc report of the third run, even though the tool was included
System information
Nextflow version: 25.04.4
Hardware: Desktop
Executor: local
OS: Ubuntu Linux
Version of nf-core/tools: 3.3.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the nf-core template code that generates the softwareVersionsToYAML().collectFile() call, then reproduce the three Nextflow commands described in the issue. Check the generated template and resume behavior; done means collected files stored outside work directories do not produce inconsistent software-version output across resumed runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100