Create frozen Conda environments for modules

Description of feature

Problem

Conda environments are not reproducible over time. The sometimes large dependency trees mean you get a different software stack next week to the one you have today. This is bad for reproducible science.

The often used workaround for this has been to use Docker images, which have the effect of freezing dependency trees, but then if you find yourself rebuilding Docker images (e.g. to patch due to security concerns) you lose those frozen dependencies. Some (e.g. Paolo, I think) would say that really, we should be using Docker as a software delivery mechanism only.

A better way of doing this is to actually record the state of the environment when modules are created, and when the conda dependencies are updated, creating a frozen dependencies file that can be used to create environments when the workflows are run.

Available solutions

pythonspeed has an excellent (if not quite up to date) summary of this.

Essentially there are two ways to go.

conda env export

Create the environments, immediately record their state.

• Advantages: no extra software required
• Disadvantages: would be difficult for developers to do on a single machine in order to generate the separate environments that would be required for e.g. MacOS and Linux. Maybe it could be done with different machines in CI?

conda-lock

See https://github.com/conda/conda-lock.

• Advantages:
  • can make multi-platform lock files
  • Bypasses the conda solver (you're basically just storing a list of URIs to the package archives). That could speed things up significantly.
• Disadvantages
  • Requires more software
  • Users would need to install conda-lock to re-create environments at run time.

How I imagine the tools commands working

I don't know how we might persuade Nextflow itself to use lock files to create the environments from lock files at run time. So imagine a different sequence:

nf-core modules conda-lock - Runs conda-lock, creates lockfiles for all architectures required
nf-core init-locked_envs - Creates environments for all the lockfiles for all the the modules of a workflow that have them.

Then, when the workflow is run, the module environments are all recognised as being in place, and off we go. This could work incrementally, such that environments were still created on the fly for modules lacking lock files.

Potential problems

• Rebuilding lock files when conda packages were bumped.
• CI to ensure the above.
• There may be some overlap with all the new funky Wave stuff

Start by tracing the existing nf-core tools command entry points and module environment handling; the issue names nf-core modules conda-lock and nf-core init-locked_envs as the proposed interfaces. Define how multi-platform lockfiles are generated, how environments are initialized, how package updates are handled in CI, and how modules without lockfiles fall back to on-the-fly creation.