nextflow-io / nextflow-io/nextflow
[Bug/Behavior] changes to Conda.cacheDir (config) prevents resumability (workflow caching)
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Bug/Feature
Expected behavior and actual behavior
Changing Conda.cacheDir in the config causes the workflow cache to miss (resumability to fail).
We prefer that nextflow resumes and reuses the existing process-results.
Any missing conda environments in the new conda.cacheDir would be recreated, since they are reproducible.
It seems the caching is based on the name of the environment path, but those are a fairly weak indicator for reproducibility.
This could optionally be gated behind the cache = "lenient" directive, but i think its preferable for most users to have caching whenever possible.
Steps to reproduce the problem
Try to resume a pipeline with cached results and conda, but a changed conda.cacheDir in the workflow config from the initial invocation.
workaround
Moving and symlinking the conda cachedir to the new location.
Environment
- Nextflow version: 25.10.0
- Java version: 21.0.9
- Operating system:Linux
- Bash version: 4.4.23
- conda: 25.9.1
Additional context
There are some circumstances like running out of storage or otherwise having to change the conda install, where this can result in the loss of days of computing time.
With the rapid growth of conda cacheDirs and limited hpc or cloud filesystems on /home, this can happen fairly quickly.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named in the report. Start by tracing the workflow resume path and the Conda.cacheDir configuration described in the reproduction steps. Done means changing Conda.cacheDir still reuses compatible cached process results while missing environments are recreated in the new location.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy, linux
- Domain
- performance, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100