nextflow-io / nextflow-io/nextflow

How to set the computing resources (cpus, memory) in total submitted jobs

Open
#5,702 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

executor/slurm good first issue stale
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Use cases:
I have a SLURM HPC with small computing resources and a few users. What I want is that when l run a nextflow pipeline that it can limit the total cpus and memory. For example, I have 10 jobs, each job requires 4 cpus and 8.GB memory. However, I want to run only with overall cpus in total is 8 cpus and 16 GB memory. As a result, it will only submit the jobs with 2 jobs. It will wait on done to submit next 4 times. However, nextflow only limits cpus and memory in each process that I can not control this.

One possible solution is using the queueSize and resourceLimits. I can set the queue sizes to be 2, the max cpus and max memory is 4. However, it takes time if I have the too many small resources processes (1 cpus, 2GB memory), it can be run 4 processes each time however, due to the queue size, it can only submit with 2 processes. Is there any alternative solution that I may miss via the nextflow documentation?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Nextflow documentation for queueSize and resourceLimits and compare their per-process behavior with the requested aggregate CPU and memory cap. Determine whether total submission limits across SLURM jobs are supported; done should be a documented configuration or a clearly scoped feature request, including behavior for mixed resource requirements.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, hpc
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.