nextflow-io / nextflow-io/nextflow
`cpus` trace value does not correspond to used logical cores of the task
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Bug report
- Relates to #6710
- Relates to #5921
Expected behavior and actual behavior
The execution trace should report the logical CPUs the process ran on, but does only the number it should have run on from the config, which is often wrong.
Steps to reproduce the problem
- Define a config with
process.resourceLimits = [cpus: 1] - Execute
nextflow run demo -r 1.0.2 -profile test,docker -c test.config --outdir out - Most of the time some task has a
%cpu> 100 which should not be, becausecpusis at1
Program output
task_id native_id name status %cpu cpus memory
3 3888223 NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE2_PE) COMPLETED 91.1% 1 4 GB
2 3888220 NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE1_PE) COMPLETED 86.0% 1 4 GB
4 3889134 NFCORE_DEMO:DEMO:FASTQC (SAMPLE2_PE) COMPLETED 76.3% 1 4 GB
1 3888837 NFCORE_DEMO:DEMO:FASTQC (SAMPLE1_PE) COMPLETED 82.2% 1 4 GB
6 3890070 NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE3_SE) COMPLETED 85.4% 1 4 GB
> 5 3890405 NFCORE_DEMO:DEMO:FASTQC (SAMPLE3_SE) COMPLETED 117.0% 1 4 GB
7 3891275 NFCORE_DEMO:DEMO:MULTIQC COMPLETED 44.3% 1 4 GB
Environment
- Nextflow version: [25.10.2]
- Operating system: Linux & MacOS
Additional context
When limiting the cpus via taskset -c <CPU_LIST> nextflow run ..., everything seems to be in order:
task_id native_id name status %cpu cpus memory
2 3329972 NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE2_PE) COMPLETED 81.7% 1 4 GB
1 3330576 NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE1_PE) COMPLETED 92.4% 1 4 GB
3 3331266 NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE3_SE) COMPLETED 90.0% 1 4 GB
4 3331898 NFCORE_DEMO:DEMO:FASTQC (SAMPLE1_PE) COMPLETED 71.5% 1 4 GB
5 3332979 NFCORE_DEMO:DEMO:FASTQC (SAMPLE2_PE) COMPLETED 73.9% 1 4 GB
6 3333940 NFCORE_DEMO:DEMO:FASTQC (SAMPLE3_SE) COMPLETED 77.1% 1 4 GB
7 3334923 NFCORE_DEMO:DEMO:MULTIQC COMPLETED 38.3% 1 4 GB
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with nextflow run demo -r 1.0.2 -profile test,docker -c test.config --outdir out and process.resourceLimits = [cpus: 1], then compare the trace's cpus value with the logical CPUs actually used. Done means the trace reports the task's used logical cores rather than only the configured CPU count, including when %cpu exceeds 100%.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100