Hang when demultiplexing on many cores with many adapters
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 587
- Forks
- 144
- Avg merge
- 2h 37m
- Merged PRs (30d)
- 1
Description
I observe a strange behavior that I never saw before when using many adapters with many cores. I have a file with 182 adapters. When I use 40 cores (or less) cutadapt starts processing (indicated by CPU usage and the status bar appearing) after 2-3 seconds. When I use 50 cores (or more) after few seconds nothing happens anymore. No CPU usage, no status bar appearing.
When I reduce the adapter than it works with 50 cores. Before I had ~140 adapter and used 80 cores without any issue.
I haved tried with cutadapt 3.4 and latest 3.7 using python 3.8.5. The machine has 3TB of RAM and 72C/144T. The workaround is just using less cores for the moment and wait a bit longer. I really like the multicore option that now works also for many adapters for a while! Thank you!
I am using pair-end mode, the adapters file is attached. The exact call of cutadapt is:
cutadapt --no-indels --no-trim --cores=50 -g file:adapt.fa -o out/trimmed-{name}_R1.fastq.gz -p out/trimmed-{name}_R2.fastq.gz input_1.fastq.gz input_2.fastq.gz
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the report with adaptersLeft.zip and the provided paired-end cutadapt command, comparing behavior below and above 50 cores. Confirm the fix by processing the attached adapter set with many cores, with CPU activity and the status bar appearing and both trimmed output files being produced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- bioinformatics
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100