nextflow-io / nextflow-io/nextflow
How to set the computing resources (cpus, memory) in total submitted jobs
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Use cases:
I have a SLURM HPC with small computing resources and a few users. What I want is that when l run a nextflow pipeline that it can limit the total cpus and memory. For example, I have 10 jobs, each job requires 4 cpus and 8.GB memory. However, I want to run only with overall cpus in total is 8 cpus and 16 GB memory. As a result, it will only submit the jobs with 2 jobs. It will wait on done to submit next 4 times. However, nextflow only limits cpus and memory in each process that I can not control this.
One possible solution is using the queueSize and resourceLimits. I can set the queue sizes to be 2, the max cpus and max memory is 4. However, it takes time if I have the too many small resources processes (1 cpus, 2GB memory), it can be run 4 processes each time however, due to the queue size, it can only submit with 2 processes. Is there any alternative solution that I may miss via the nextflow documentation?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Nextflow documentation for queueSize and resourceLimits and compare their per-process behavior with the requested aggregate CPU and memory cap. Determine whether total submission limits across SLURM jobs are supported; done should be a documented configuration or a clearly scoped feature request, including behavior for mixed resource requirements.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems, hpc
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100