llnl / llnl/ATS

Slurm does not honor --nodelist

Open
#173 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
10
Forks
6
PR merge metrics
No merged PRs in 30d

Description

We use the --nodelist option to inform slurm it must run a job on a specific hardware node.

Slurm honors the request in **most**, but not all, cases. This behavior seems like slurm takes the --nodelist as a suggestion rather than a requirement. In heavy use cases, slurm seems do fit the job wherever it can.

From the HelloSameNode test directory, the slurm testing shows just 1 test case where the requirement is not honored. It is repeatable.

The log shows the --nodelist=rzwhippet69 requirement. But the code was run on rzwhippet68.

#ATS:commandLine =srun --comment="nompitype" --label -J t60_23test_file_write_t23_label_072717 --comment=nopartition --exclusive --comment="nounbuffered" --mpi
bind=off --comment=no_distribution --nodes=1-3 --cpus-per-task=1 -**-nodelist=rzwhippet69** --ntasks=60 --comment=nosargs /usr/WS2/dawson/Git-ATS-GitHub-SameNode/t
est/HelloSameNode/test_file_write.py nodeB_23


Writing a file n /usr/tmp on node rzwhippet68

We will need to work with the Slurm developers to resolve this issue.

For comparison, the equivalent option in Flux always works, flux takes the hardware node as a requirement.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the HelloSameNode test directory, especially test_file_write.py, and reproduce the failing srun command with --nodelist=rzwhippet69. Compare the requested node with the reported execution node; done means the test consistently runs on the requested hardware node, likely requiring coordination with Slurm developers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
hpc, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.