nf-core / nf-core/modules

Bowtie2 + Samtools pipeline in nf-core module may use double the intended threads

Open
#9,401 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Nextflow
Stars
429
Forks
1.1k
Avg merge
1d 6h
Merged PRs (30d)
153

Description

Have you checked the docs?
Description of the bug

In the nf-core Bowtie2 alignment module, the pipeline runs something like:

bowtie2 ... --threads $task.cpus ... | samtools ... --threads $task.cpus ...

see https://github.com/nf-core/modules/blob/fa1477128e0ff24675c8f56d6c334fcc323025c4/modules/nf-core/bowtie2/align/main.nf#L57-L64

Because Bowtie2 and Samtools are piped, both programs seem to independently use $task.cpus threads.

I observed this when I was running nf-core/atacseq, and the bowtie_align step was using more cores than it is supposed to use. When inspecting processes with htop, I saw a bowtie2 process and a samtools process, both using 12 threads, and I had only given 12 threads to this process. I scheduled the process with slurm on our cluster.

Should this be solved by making sure that bowtie2 and samtools both ask for less threads than $task.cpus?

Command used and terminal output

Relevant files

No response

System information

nextflow 25.10.0
run on our HPC with slurm executor
container engine: singularity
CentOS Linux release 7.4
running the dev branch of nf-core/atacseq, which uses the bowtie2_align module

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read modules/nf-core/bowtie2/align/main.nf around lines 57-64, then inspect how the module assigns task.cpus to Bowtie2 and Samtools. Reproduce or verify the behavior through the nf-core/atacseq bowtie_align step under Slurm, and confirm that the module's total thread usage matches its requested allocation.

Written by the indexing model from the issue text.

Assessment

Domain
bioinformatics, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.