mlr-org / mlr-org/batchtools

max.concurrent.jobs issue with SLURM arrays

Open
#198 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
184
Forks
53
Avg merge
7d 2h
Merged PRs (30d)
1

Description

Hi there -- this should be an easy fix -- it looks like when resources is set with max.concurrent.jobs=N, and the job is run as a SLURM array job, it is not really working as expected, since if you lump, say, 1000 array tasks into the job, and only one job is submitted, you'll (practically speaking) get up to 1000 concurrent "jobs" (tasks in SLURM-speak, but really the same thing).

I'd like to suggest that if batchtools is running via SLURM and an array job, max.concurrent.jobs modifies the array setting properly following:
https://slurm.schedmd.com/job_array.html

All you'd need to do is pass that resource variable to the line (in your .tmpl):
<%= if (array.jobs) sprintf("#SBATCH --array=1-%i", nrow(jobs)) else "" %>

to be something like:
<%= if (array.jobs) sprintf("#SBATCH --array=1-%i'%'%max.concurrent.jobs", nrow(jobs)) else "" %>

where max.concurrent is the number of concurrent files passed from batchtools resource, and note that it needs to be a "%" so I guess you need a "true" % in the string.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the SLURM array job .tmpl file and inspect how the batchtools resource value for max.concurrent.jobs is passed into the array directive. Check the SLURM job-array documentation linked in the issue, then verify that array submissions enforce the requested concurrency limit without changing non-array behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
hpc
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.