Document Recommended Practices for High-Throughput Slurm Workloads
@Premas is already working on this.
Since Sep 3, 2026.
- Dominant language
- Python
- Stars
- 24
- Forks
- 15
- Avg merge
- 10d 3h
- Merged PRs (30d)
- 1
Description
What would you like to see added?
Users running Nextflow/nf-core workflows may submit a large number of jobs in a short period of time, which can create a high volume of job submission requests to the Slurm controller. For such cases, reducing the submission rate help prevent large bursts of requests and reduce the risk of temporarily overloading the Slurm controller.
We do not currently have a documented limit on the number of job submissions allowed within a given time period for this type of workflow. However, we recommend users avoid submitting large bursts of jobs that may impact scheduler responsiveness.
Proposed Guidance
Recommend users configure Nextflow to limit the job submission rate:
executor {
submitRateLimit = '1/sec'
}
This allows Nextflow to submit a maximum of one job per second (up to 60 job submissions per minute).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.