nextflow-io / nextflow-io/nextflow
Conda env not removed when failing in pip stage
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Bug report
Expected behavior and actual behavior
Conda env is correctly removed after failing in conda env creation stage. However, if it fails in pip installation stage (can be a network error for example), the error is displayed and the pipeline run fails as expected, but the environment is kept. So in subsequent runs, user can get confusing error messages that a certain package does not exist, since it runs in the failed environment.
Steps to reproduce the problem
Use this conda env yml:
channels:
- nodefaults
- conda-forge
dependencies:
- python = 3.11
- pip
- pip:
- nonexistentpackage
Program output
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/Users/prihodad/Library/Application Support/ovo/workdir/work/conda/env-56ad78d602a61cd3-4ef8249679916ddf5a4ded787317a380/bin/python', '-m', 'pip', 'install', '-U', '-r', '/Users/prihodad/Documents/projects/ovo/ovo/pipeline/containers/boltz/condaenv.wn99havt.requirements.txt', '--exists-action=b']
Pip subprocess output:
! ! ! Warning ! ! !
You are using an experimental build of OpenMM v7.5.1.
This is NOT SUITABLE for production!
It has not been properly tested on this platform and we cannot guarantee it provides accurate results.
failed
==> WARNING: A newer version of conda exists. <==
current version: 22.11.1
latest version: 25.7.0
Please update conda by running
$ conda update -n base -c conda-forge conda
Or to minimize the number of packages updated during conda update use
conda install conda=25.7.0
Pip subprocess error:
ERROR: Could not find a version that satisfies the requirement nonexistentpackage (from versions: none)
ERROR: No matching distribution found for nonexistentpackage
CondaEnvException: Pip failed
-- Check '.nextflow.log' file for details
Environment
- Nextflow version: any
- Java version: any
- Operating system: any
- Bash version: any
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the supplied conda environment YAML and inspect .nextflow.log for the pip-stage error. Trace the conda environment lifecycle around that failure; done means a failed pip installation removes the environment so subsequent runs do not reuse it, while cleanup after conda creation failures still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100