marcelm / marcelm/cutadapt

Hang when demultiplexing on many cores with many adapters

Open
#613 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
587
Forks
144
Avg merge
2h 37m
Merged PRs (30d)
1

Description

I observe a strange behavior that I never saw before when using many adapters with many cores. I have a file with 182 adapters. When I use 40 cores (or less) cutadapt starts processing (indicated by CPU usage and the status bar appearing) after 2-3 seconds. When I use 50 cores (or more) after few seconds nothing happens anymore. No CPU usage, no status bar appearing.

When I reduce the adapter than it works with 50 cores. Before I had ~140 adapter and used 80 cores without any issue.

I haved tried with cutadapt 3.4 and latest 3.7 using python 3.8.5. The machine has 3TB of RAM and 72C/144T. The workaround is just using less cores for the moment and wait a bit longer. I really like the multicore option that now works also for many adapters for a while! Thank you!

I am using pair-end mode, the adapters file is attached. The exact call of cutadapt is:
cutadapt --no-indels --no-trim --cores=50 -g file:adapt.fa -o out/trimmed-{name}_R1.fastq.gz -p out/trimmed-{name}_R2.fastq.gz input_1.fastq.gz input_2.fastq.gz

adaptersLeft.zip

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the report with adaptersLeft.zip and the provided paired-end cutadapt command, comparing behavior below and above 50 cores. Confirm the fix by processing the attached adapter set with many cores, with CPU activity and the status bar appearing and both trimmed output files being produced.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
bioinformatics
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.