nextflow-io / nextflow-io/nextflow

Using "docker.enabled = true" affects availability of conda command in conda-based wave containers

Open
#6,439 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

software/conda software/wave stale
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

Expected behavior and actual behavior

We use a mix of public Docker images and YAML-defined conda environments. It appears that enabling Docker (docker.enabled = true) makes the "conda" command in yaml-defined conda environments unavailable. This is unexpected behavior to me. I was hoping to use the "conda" command to e.g. retrieve additional information about the actually installed packages in the YAML-defined environment.

Steps to reproduce the problem

With Docker disabled:

main.nf

conda.enabled = true
//docker.enabled = true
wave.enabled = true
wave.strategy = 'conda,container'

process {
    withLabel: 'conda_samtools' {
        conda = "$projectDir/samtools.yaml"
    }
}

nextflow.config

process test_conda {
    label 'conda_samtools'
    debug = true
    script:
    """
    echo "Conda env"
    samtools --version-only
    echo "---"
    """
}

process test_conda2 {
    label 'conda_samtools'
    debug = true
    script:
    """
    echo "Use conda"
    conda list samtools
    echo "---"
    """
}

workflow {
    test_conda()
    test_conda2()
}

With Docker enabled:

main.nf

conda.enabled = true
docker.enabled = true
wave.enabled = true
wave.strategy = 'conda,container'

process {
    withLabel: 'conda_samtools' {
        conda = "$projectDir/samtools.yaml"
    }
}
Program output

With Docker disabled:

 N E X T F L O W   ~  version 25.04.7

Launching `/main.nf` [jovial_salas] DSL2 - revision: 819790a717

executor >  local (2)
[ad/a2b1cc] process > test_conda  [100%] 1 of 1 ?
[d7/26b866] process > test_conda2 [100%] 1 of 1 ?
Conda env
1.22.1+htslib-1.22.1
---

Use conda
# packages in environment at /nextflow/work/conda/env-0937c70d5d6fa28d-700d5afef5901d27aac78c3b847fed0e:
#
# Name                    Version                   Build  Channel
samtools                  1.22.1               h96c455f_0    bioconda

With Docker enabled:

 N E X T F L O W   ~  version 25.04.7

Launching `/main.nf` [exotic_carson] DSL2 - revision: cec9713e8c

executor >  local (2)
[75/5dffc6] process > test_conda  [100%] 1 of 1 ?
[8f/480362] process > test_conda2 [  0%] 0 of 1
Conda env
1.22.1+htslib-1.22.1
---
executor >  local (2)
[75/5dffc6] process > test_conda  [100%] 1 of 1 ?
[8f/480362] process > test_conda2 [  0%] 0 of 1 ?
Conda env
1.22.1+htslib-1.22.1
---

ERROR ~ Error executing process > 'test_conda2'

Caused by:
  Process `test_conda2` terminated with an error exit status (127)


Command executed:

  echo "Use conda"
  conda list samtools
  echo "---"

Command exit status:
  127

Command output:
  Use conda

Command error:
  Unable to find image 'wave.seqera.io/wt/c0f876d019bd/wave/build:samtools-1.22.1--0a2cf3e789c10533' locally
  samtools-1.22.1--0a2cf3e789c10533: Pulling from wt/c0f876d019bd/wave/build
  Digest: sha256:a552e47e0624c4967635df07ef1569d899300f77f091d5bc8dabeca6019fa6ce
  Status: Downloaded newer image for wave.seqera.io/wt/c0f876d019bd/wave/build:samtools-1.22.1--0a2cf3e789c10533
  Use conda
  .command.sh: line 3: conda: command not found

Work dir:
  /nextflow/work/8f/480362a81b65ffc1a001a92e27168f

Container:
  wave.seqera.io/wt/c0f876d019bd/wave/build:samtools-1.22.1--0a2cf3e789c10533

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details
Environment
  • Nextflow version: 25.04.7
  • Java version: openjdk 17.0.10-internal 2024-01-16
  • Operating system: Linux
  • Bash version: GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu)
Additional context

(Add any other context about the problem here)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the two configurations in main.nf and nextflow.config, comparing the commands run with Docker disabled and enabled. Inspect the reported .nextflow.log and container details to trace why conda list fails with exit status 127. Done means the YAML-defined conda environment still permits the conda command when Docker and Wave are enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, groovy
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.