nextflow-io / nextflow-io/nextflow

[Bug/Behavior] changes to Conda.cacheDir (config) prevents resumability (workflow caching)

Open
#6,613 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

software/conda stale
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug/Feature

Expected behavior and actual behavior

Changing Conda.cacheDir in the config causes the workflow cache to miss (resumability to fail).
We prefer that nextflow resumes and reuses the existing process-results.
Any missing conda environments in the new conda.cacheDir would be recreated, since they are reproducible.

It seems the caching is based on the name of the environment path, but those are a fairly weak indicator for reproducibility.

This could optionally be gated behind the cache = "lenient" directive, but i think its preferable for most users to have caching whenever possible.

Steps to reproduce the problem

Try to resume a pipeline with cached results and conda, but a changed conda.cacheDir in the workflow config from the initial invocation.

workaround

Moving and symlinking the conda cachedir to the new location.

Environment
  • Nextflow version: 25.10.0
  • Java version: 21.0.9
  • Operating system:Linux
  • Bash version: 4.4.23
  • conda: 25.9.1
Additional context

There are some circumstances like running out of storage or otherwise having to change the conda install, where this can result in the loss of days of computing time.
With the rapid growth of conda cacheDirs and limited hpc or cloud filesystems on /home, this can happen fairly quickly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named in the report. Start by tracing the workflow resume path and the Conda.cacheDir configuration described in the reproduction steps. Done means changing Conda.cacheDir still reuses compatible cached process results while missing environments are recreated in the new location.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy, linux
Domain
performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.