nf-core / nf-core/tools

Template: Resume function can lead to inconsistencies in software version reporting

Open
#3,653 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
322
Forks
255
Avg merge
2d 3h
Merged PRs (30d)
5

Description

Description of the bug

The current nf-core template, software versions are collected without disabling the cache function of .collectFile():

    // Collate and save software versions
    softwareVersionsToYAML(ch_versions)
        .collectFile(
            storeDir: "${params.outdir}/pipeline_info",
            name: 'nf_core_'  +  'diseasemodulediscovery_software_'  + 'mqc_'  + 'versions.yml',
            sort: true,
            newLine: true
        ).set { ch_collated_versions }

This can lead to inconsistencies (missing/additional processes listed) in combination with multiple pipeline runs using the Nextflow resume function.

A fix would be to disable the collectFile caching:

    // Collate and save software versions
    softwareVersionsToYAML(ch_versions)
        .collectFile(
            storeDir: "${params.outdir}/pipeline_info",
            name: 'nf_core_'  +  'diseasemodulediscovery_software_'  + 'mqc_'  + 'versions.yml',
            sort: true,
            newLine: true,
            cache: false
        ).set { ch_collated_versions }

Or to not provide a storeDir, but cache the file in the work directory instead.
Something similar can happen to all files created using collectFile, which are stored outside of the work directory with caching enabled.

Command used and terminal output
# run pipeline (first run)
nextflow run diseasemodulediscovery -r dev -profile test,docker --outdir results

# delete result directory
rm -r results

# run pipeline again with resume, but skip a step (second run)
nextflow run diseasemodulediscovery -r dev -profile test,docker --outdir results --skip_gprofiler -resume

# run full pipeline again with resume (third run)
nextflow run diseasemodulediscovery -r dev -profile test,docker --outdir results -resume

# the third run uses the caching of gprofiler from the first run (cached in work), but the caching of the software versions from the second run (cached in results)
# As a result, the software versions of gprofiler are not listed in the multiqc report of the third run, even though the tool was included
System information

Nextflow version: 25.04.4
Hardware: Desktop
Executor: local
OS: Ubuntu Linux
Version of nf-core/tools: 3.3.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the nf-core template code that generates the softwareVersionsToYAML().collectFile() call, then reproduce the three Nextflow commands described in the issue. Check the generated template and resume behavior; done means collected files stored outside work directories do not produce inconsistent software-version output across resumed runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.