nextflow-io / nextflow-io/nextflow

Conda env not removed when failing in pip stage

Open
#6,437 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

software/conda stale
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

Expected behavior and actual behavior

Conda env is correctly removed after failing in conda env creation stage. However, if it fails in pip installation stage (can be a network error for example), the error is displayed and the pipeline run fails as expected, but the environment is kept. So in subsequent runs, user can get confusing error messages that a certain package does not exist, since it runs in the failed environment.

Steps to reproduce the problem

Use this conda env yml:

channels:
  - nodefaults
  - conda-forge
dependencies:
  - python = 3.11
  - pip
  - pip:
      - nonexistentpackage
Program output

      Collecting package metadata (repodata.json): ...working... done
      Solving environment: ...working... done
      
      Downloading and Extracting Packages
      
      Preparing transaction: ...working... done
      Verifying transaction: ...working... done
      Executing transaction: ...working... done
      Installing pip dependencies: ...working... Ran pip subprocess with arguments:
      ['/Users/prihodad/Library/Application Support/ovo/workdir/work/conda/env-56ad78d602a61cd3-4ef8249679916ddf5a4ded787317a380/bin/python', '-m', 'pip', 'install', '-U', '-r', '/Users/prihodad/Documents/projects/ovo/ovo/pipeline/containers/boltz/condaenv.wn99havt.requirements.txt', '--exists-action=b']
      Pip subprocess output:
      
      ! ! ! Warning ! ! !
      
      You are using an experimental build of OpenMM v7.5.1.
      This is NOT SUITABLE for production!
      It has not been properly tested on this platform and we cannot guarantee it provides accurate results.
      
      
      
      failed
      
      
      ==> WARNING: A newer version of conda exists. <==
        current version: 22.11.1
        latest version: 25.7.0
      
      Please update conda by running
      
          $ conda update -n base -c conda-forge conda
      
      Or to minimize the number of packages updated during conda update use
      
           conda install conda=25.7.0
      
      
      Pip subprocess error:
      ERROR: Could not find a version that satisfies the requirement nonexistentpackage (from versions: none)
      ERROR: No matching distribution found for nonexistentpackage
      
      
      CondaEnvException: Pip failed



 -- Check '.nextflow.log' file for details
Environment
  • Nextflow version: any
  • Java version: any
  • Operating system: any
  • Bash version: any
Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the supplied conda environment YAML and inspect .nextflow.log for the pip-stage error. Trace the conda environment lifecycle around that failure; done means a failed pip installation removes the environment so subsequent runs do not reuse it, while cleanup after conda creation failures still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.