llnl / llnl/maestrowf

SLURM: specifying extra arguments for GPU binding

Open
#436 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
161
Forks
49
PR merge metrics
No merged PRs in 30d

Description

Is there a recommended way to specify extra SLURM options for GPU bindings?

I tried using the `args:` batch block key (https://maestrowf.readthedocs.io/en/latest/Maestro/scheduling.html), but the options did not get propagated to the *.sh job script.

Following https://github.com/LLNL/maestrowf/issues/340, the workaround I've used so far is to specify these options as part of the run command so that they get copied into the job script:
```
- name: run-sim
description: Run the simulation.
run:
cmd: |
#SBATCH --mem=0
#SBATCH --constraint="scratch"
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=16
#SBATCH --gpus-per-task=1
#SBATCH --gpu-bind=none

srun bash -c "
export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID));
$(BINARY_PATH) -i $(generate-infile.workspace)/params.in" > logfile.txt
depends: [generate-infile]
nodes: 1
exclusive: True
walltime: "00:10:00"
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how the `args:` batch block key is handled when the generated `*.sh` job script is built. Compare it with the documented run-command workaround and verify that extra SLURM options, including GPU-binding options, are propagated into the script.

Written by the indexing model from the issue text.

Assessment

Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.