Pipeline download purpose of cache-utilization parameters after refactoring
Open
@JulianFlesch is already working on this.
Since Aug 27, 2025.
download
- Dominant language
- Python
- Stars
- 322
- Forks
- 255
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 5
Description
Note: This was commented on main the downloads functionality pull request https://github.com/nf-core/tools/pull/3634
To comment on https://github.com/nf-core/tools/pull/3634#discussion_r2195332147 and https://github.com/nf-core/tools/pull/3634#discussion_r2223736610 , here are my observations from testing combinations of cache/library parameters and options.
- The proposed implementation/purpose of
-u copyis to create a standalone copy of the pipeline. Thus, its localsingularity-images/directory must be complete, andnextflow.configmust be modified to point at it. The pipeline can then run without a cache/library directory externally set.
Rightly, the command takes images from the pre-existing cache or library if possible, rather than downloading absolutely everything.
The current implementation also copies images that are downloaded to the cache directory. This has no runtime effect but honours the purpose of the cache/library: after the command is run, they together hold all images as well, thus helping future / other invocations of Nextflow. Note that the cache may not hold all images itself, as some may rather be in the library only. - The proposed implementation/purpose of
-u amendis to create a copy of the pipeline that entirely relies on the external cache alone. That's why the localsingularity-images/directory is left empty and all images are rather deposited into the cache.nextflow.configis not modified.
Here also, the command only downloads the images that are not present in the cache/library.
Contrary to-u copy, images are copied to the cache to make it complete on its own, rather than allowing some images to be in the library only. - However I don't understand the implementation/purpose of skipping the
-uoption. When I runnf-core pipelines downloadwithout-u, the localsingularity-images/directory is made complete butnextflow.configisn't updated, meaningsingularity-images/is not used. Fortunately, like-u copy, the cache is updated, so the pipeline can still run offline provided the cache (and the library) are set externally. It's essentially something in between-u copyand-u amend, but I can't define the rationale.
My thinking is that:
- In
-u amendmode, no need to duplicate images from the library to the cache. In other words, assume that the pipeline will run with both the cache and library set the same way as innf-core pipelines download, rather than only the cache. - In
-u amendmode, no need to create a localsingularity-images/directory if it's left empty. - Don't allow
-uto be unset ?
Originally posted by @muffato in https://github.com/nf-core/tools/issues/3634#issuecomment-3193885334
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.