huggingface / huggingface/datatrove

solved: how to launch a slurm executor from an interactive slurm job

Open
#248 0 comments 6 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

I forget where I saw it in the docs/code where it said not to launch a slurm executor from an `srun` interactive session - which is not quite always possible.

There is a simple workaround - unset `SLURM_*` env vars and then launch and it works just fine.

```
unset $(printenv | grep SLURM | sed -E 's/(.*)=.*/\1/' | xargs)
./my_datatrove_slurm.py
```
Of course, your `srun` session will now be w/o its env vars - which you may or may not care for.

To help others to find the solution, the error is likely to be:

```
srun: error: CPU binding outside of job step allocation, allocated CPUs are: 0x0000000000000FFF80000000000000000000000FFF8000000000.
srun: error: Task launch for StepId=120986.0 failed on node xxx-yyy-11: Unable to satisfy cpu bind request
srun: error: Application launch failed: Unable to satisfy cpu bind request
srun: Job step aborted
```

There is also [this discussion](https://groups.google.com/g/slurm-users/c/mp_JRutKmCc) that proposes to unset just `SLURM_CPU_BIND_*` env vars, so you'd then:

```
unset $(printenv | grep SLURM_CPU_BIND | sed -E 's/(.*)=.*/\1/' | xargs)
./my_datatrove_slurm.py
```

If you want to unset them just for the datatrove launcher use this one-liner syntax
```
SLURM_CPU_BIND= SLURM_CPU_BIND_VERBOSE= SLURM_CPU_BIND_LIST= SLURM_CPU_BIND_TYPE= ./my_datatrove_slurm.py
```
or you could of course unset them inside your script as well, which would make the launching even simpler.

That way all `SLURM_*` env vars will remain intact in your shell environment if you need them for something else.

edit:

I added:

```
import os
# datatrove fails to start slurm jobs from an interactive slurm job,
# so hack to pretend we aren't inside an interactive slurm job by removing SLURM env vars
for key in os.environ.keys():
if key.startswith("SLURM_"):
os.environ.pop(key)
```
on top of my script to make it always work.

Contributor guide

No contributing guide indexed for this repository

Research direction

No file, test, or entry point is named. Start by locating the Slurm launcher documentation or implementation and verify the interactive-job failure and environment-variable workaround described here. Done means making the workaround discoverable for users launching from an interactive Slurm session.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.