Tracking of Conda environments as a dependency

New feature

I may be misdiagnosing my problems but it's my belief that Nextflow doesn't do any tracking of its conda environments beside the path to the environment. I work in an HPC environment where I'd like all working files to be put on a scratch filesystem and publishing the useful artifacts to my permanent project directory, which Nextflow provides excellent support for. However, I have noticed that commands often fail due to programs going missing in my Conda environments, I believe this is due to the scratch disk cleaning up components of my environment after a period of no access.

It feels to me like Nextflow should ensure that the conda environments it creates via the conda "pkg" directive should have some tracking such that: if the conda environment is changed, then it should recreate it.

Suggested implementation

I imagine you could hash environments, if hash changes then create a new environment with the directive, if the new environment is a hash match then nothing happens, if hash changes then all processes that use that environment must be rerun.

Though I think the number of files in the average conda environment might make this prohibitive as a default, so perhaps it can be left as an optional check?

Start by tracing the conda handling behind the conda "pkg" directive and how Nextflow tracks process dependencies. Determine how an environment change could be detected without hashing every file, and how that state would affect process reruns. Done means a defined, tested behavior for optional environment tracking and recreation.