Lightning-AI / Lightning-AI/pytorch-lightning
slurm env incorrectly complains about srun with salloc interactive session.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
I get an error message (see below) complaining that I'm not using srun, preventing me from running my parallel code. But in reality I did use srun, but I'm also in interactive mode with salloc. In order to get past this error I had to modify the file: `/u/ddeighan/miniforge3/envs/uqops/lib/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py` to manually disable checking if the allocation was in interactive mode.
**There needs to be better sanity checking code to tell the difference between `srun -n1 --pty bash` (invalid, not really using srun) & `salloc ...; srun python train.py`** (valid, using srun properly)
### What version are you seeing the problem on?
v2.5
### How to reproduce the bug
salloc -N2 --gpus=2 ...
srun python train_parallel.py
### Error messages and logs
/u/ddeighan/miniforge3/envs/uqops/lib/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py:204: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python channel.py ...
### Environment
Current environment
```
#- PyTorch Lightning Version (e.g., 2.5.0):
#- PyTorch Version (e.g., 2.5):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
```
### More info
_No response_
cc @lantiga
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Inspect lightning_fabric/plugins/environments/slurm.py around the check reported at line 204. Reproduce the salloc followed by srun command from the report and compare it with srun -n1 --pty bash; done means the valid interactive allocation no longer triggers the warning while the invalid case is still detected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100