Lightning-AI / Lightning-AI/pytorch-lightning
Using srun on a SLURM cluster causes the job to exit prematurely, even with auto_requeue=False
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
[This assertion](https://github.com/Lightning-AI/pytorch-lightning/blob/a6562b4ae7623acb87f35d49216f90aae2363b18/src/lightning/fabric/plugins/environments/slurm.py#L220) raises when inside SRUN using multiple tasks, even after doing adding `SLURMEnvironment(auto_requeue=False)`. The jobs are initially submitted through submitit-slurm.
Even after: `os.environ["SLURM_NTASKS_PER_NODE"] = os.environ["SLURM_TASKS_PER_NODE"]`, another error occurs.
Also following the advice from this thread (#6389) to do the following does not work. See also #15709.
```
from lightning.pytorch.plugins.environments import SLURMEnvironment
SLURMEnvironment.detect = lambda: False
trainer = Trainer(...)
```
There should be a simple and unambiguous way to disable _all_ special SLURM handling in lightning.
After making `_validate_srun_used`, `_validate_srun_variables`, `_is_srun_used`, `_is_slurm_interactive_mode`, all return False/early, my code runs fine! There should be a simple `disable_slurm_detection` flag, ideally also configurable as an environment variable
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/lightning/fabric/plugins/environments/slurm.py at the _validate_srun_used assertion and review _validate_srun_variables, _is_srun_used, and _is_slurm_interactive_mode. Define a clear way to disable all special SLURM handling, including the requested configuration or environment-variable behavior, so the submitit-slurm job can run without these checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100