huggingface / huggingface/datatrove
solved: how to launch a slurm executor from an interactive slurm job
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
I forget where I saw it in the docs/code where it said not to launch a slurm executor from an `srun` interactive session - which is not quite always possible.
There is a simple workaround - unset `SLURM_*` env vars and then launch and it works just fine.
```
unset $(printenv | grep SLURM | sed -E 's/(.*)=.*/\1/' | xargs)
./my_datatrove_slurm.py
```
Of course, your `srun` session will now be w/o its env vars - which you may or may not care for.
To help others to find the solution, the error is likely to be:
```
srun: error: CPU binding outside of job step allocation, allocated CPUs are: 0x0000000000000FFF80000000000000000000000FFF8000000000.
srun: error: Task launch for StepId=120986.0 failed on node xxx-yyy-11: Unable to satisfy cpu bind request
srun: error: Application launch failed: Unable to satisfy cpu bind request
srun: Job step aborted
```
There is also [this discussion](https://groups.google.com/g/slurm-users/c/mp_JRutKmCc) that proposes to unset just `SLURM_CPU_BIND_*` env vars, so you'd then:
```
unset $(printenv | grep SLURM_CPU_BIND | sed -E 's/(.*)=.*/\1/' | xargs)
./my_datatrove_slurm.py
```
If you want to unset them just for the datatrove launcher use this one-liner syntax
```
SLURM_CPU_BIND= SLURM_CPU_BIND_VERBOSE= SLURM_CPU_BIND_LIST= SLURM_CPU_BIND_TYPE= ./my_datatrove_slurm.py
```
or you could of course unset them inside your script as well, which would make the launching even simpler.
That way all `SLURM_*` env vars will remain intact in your shell environment if you need them for something else.
edit:
I added:
```
import os
# datatrove fails to start slurm jobs from an interactive slurm job,
# so hack to pretend we aren't inside an interactive slurm job by removing SLURM env vars
for key in os.environ.keys():
if key.startswith("SLURM_"):
os.environ.pop(key)
```
on top of my script to make it always work.
Contributor guide
No contributing guide indexed for this repository
Research direction
No file, test, or entry point is named. Start by locating the Slurm launcher documentation or implementation and verify the interactive-job failure and environment-variable workaround described here. Done means making the workaround discoverable for users launching from an interactive Slurm session.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100