nf-core / nf-core/tools

MultiQC results not cacheable

Open
#3,110 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
322
Forks
255
Avg merge
2d 3h
Merged PRs (30d)
5

Description

Description of the bug

Within the demux pipeline, we observe that MultiQC is never cached and thus always reexecuted, see also https://github.com/nf-core/demultiplex/issues/233 for a full discussion. Some pipelines configure in the conf/modules.config that MultiQC should not be cached at all (by setting caching = false), whereas this is not desired and thus also not configured in demultiplex as such.

The main issue boils down to some files being created using the official nf-core tools template tooling, e.g. workflow_summary_mqc.yaml and methods_description_mqc.yaml by changing whats ingested into these two YAML files as well as the sorting of these files / content that is in these files. The latter can be addressed by some channel magic, the former is more difficult to address and should thus be tackled probably here within the template and then auto-fixed via a template merge across multiple pipelines.

If people then actively turn off caching thats their choice - but at the moment its broken for those who would like not to re-run steps with MultiQC involved unfortunately :)

Command used and terminal output
nf-core run nf-core/demultiplexing -profile test,docker --outdir fancy_dir
System information

Doesnt really matter here

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the nf-core tools template code that generates workflow_summary_mqc.yaml and methods_description_mqc.yaml, then review the linked demultiplex discussion. Compare how the YAML contents and ordering affect MultiQC caching, and identify the template-level change needed so generated results remain cacheable without forcing caching=false across pipelines.

Written by the indexing model from the issue text.

Assessment

Domain
bioinformatics, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.