nf-core / nf-core/tools

Pipeline download purpose of cache-utilization parameters after refactoring

Open
#3,731 3 comments 0 reactions 1 assignee View on GitHub

@JulianFlesch is already working on this.

Since Aug 27, 2025.

download
Dominant language
Python
Stars
322
Forks
255
Avg merge
2d 3h
Merged PRs (30d)
5

Description

Note: This was commented on main the downloads functionality pull request https://github.com/nf-core/tools/pull/3634

To comment on https://github.com/nf-core/tools/pull/3634#discussion_r2195332147 and https://github.com/nf-core/tools/pull/3634#discussion_r2223736610 , here are my observations from testing combinations of cache/library parameters and options.

  1. The proposed implementation/purpose of -u copy is to create a standalone copy of the pipeline. Thus, its local singularity-images/ directory must be complete, and nextflow.config must be modified to point at it. The pipeline can then run without a cache/library directory externally set.
    Rightly, the command takes images from the pre-existing cache or library if possible, rather than downloading absolutely everything.
    The current implementation also copies images that are downloaded to the cache directory. This has no runtime effect but honours the purpose of the cache/library: after the command is run, they together hold all images as well, thus helping future / other invocations of Nextflow. Note that the cache may not hold all images itself, as some may rather be in the library only.
  2. The proposed implementation/purpose of -u amend is to create a copy of the pipeline that entirely relies on the external cache alone. That's why the local singularity-images/ directory is left empty and all images are rather deposited into the cache. nextflow.config is not modified.
    Here also, the command only downloads the images that are not present in the cache/library.
    Contrary to -u copy, images are copied to the cache to make it complete on its own, rather than allowing some images to be in the library only.
  3. However I don't understand the implementation/purpose of skipping the -u option. When I run nf-core pipelines download without -u, the local singularity-images/ directory is made complete but nextflow.config isn't updated, meaning singularity-images/ is not used. Fortunately, like -u copy, the cache is updated, so the pipeline can still run offline provided the cache (and the library) are set externally. It's essentially something in between -u copy and -u amend, but I can't define the rationale.

My thinking is that:

  1. In -u amend mode, no need to duplicate images from the library to the cache. In other words, assume that the pipeline will run with both the cache and library set the same way as in nf-core pipelines download, rather than only the cache.
  2. In -u amend mode, no need to create a local singularity-images/ directory if it's left empty.
  3. Don't allow -u to be unset ?

Originally posted by @muffato in https://github.com/nf-core/tools/issues/3634#issuecomment-3193885334

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.