nextflow-io / nextflow-io/nextflow

conda + pip install create tmp files in yml file path

Open
#5,656 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

software/conda stale
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

Expected behavior and actual behavior

when nextflow run use conda yml file which contains pip, (eg https://www.nextflow.io/docs/latest/conda.html#python-packages-from-pypi). The pip create tmp files (requirements.txt) in worflow dir under config dir ($baseDir/config/condaenv.sobah5ol.requirements.txt).

The problem only happens when pip is used in conda yml

For local pipeline, where only dev team has write access, downstream users of such pipeline without write permission to $baseDir worflow dir, will fail to run pipeline.

using conda directly install rather than pip install solved the error, but I hope to find a systematic fix

name: my-env-2
channels:
  - conda-forge
  - bioconda
dependencies:
  - numpy
Steps to reproduce the problem

use the official example, but make the worflow dir 744 or drwxr--r--, when execute from user account in the same group, will has write permission errors.
(eg https://www.nextflow.io/docs/latest/conda.html#python-packages-from-pypi)

name: my-env-2
channels:
  - conda-forge
  - bioconda
dependencies:
  - pip
  - pip:
    - numpy
    - pandas
    - matplotlib
Program output

(Copy and paste here output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the .nextflow.log file.)

Environment
  • Nextflow version: [?]
  • Java version: [?]
  • Operating system: [Linux]
  • Bash version: (use the command $SHELL --version)
Additional context

I think the issue description is sufficient, if more details are needed, please let me know. Thank you.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the nextflow run conda environment setup using the official Python-packages-from-PyPI example, and reproduce it with the workflow directory made non-writable. Trace where condaenv.sobah5ol.requirements.txt is created; done means pip-backed conda environments run successfully without writing temporary requirements files under $baseDir.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy, python
Domain
devops, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.