Lightning-AI / Lightning-AI/pytorch-lightning

slurm env incorrectly complains about srun with salloc interactive session.

Open
#20,776 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug environment: slurm ver: 2.5.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

I get an error message (see below) complaining that I'm not using srun, preventing me from running my parallel code. But in reality I did use srun, but I'm also in interactive mode with salloc. In order to get past this error I had to modify the file: `/u/ddeighan/miniforge3/envs/uqops/lib/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py` to manually disable checking if the allocation was in interactive mode.

**There needs to be better sanity checking code to tell the difference between `srun -n1 --pty bash` (invalid, not really using srun) & `salloc ...; srun python train.py`** (valid, using srun properly)

### What version are you seeing the problem on?

v2.5

### How to reproduce the bug

salloc -N2 --gpus=2 ...
srun python train_parallel.py

### Error messages and logs

/u/ddeighan/miniforge3/envs/uqops/lib/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py:204: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python channel.py ...

### Environment

Current environment

```
#- PyTorch Lightning Version (e.g., 2.5.0):
#- PyTorch Version (e.g., 2.5):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
```

### More info

_No response_

cc @lantiga

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Inspect lightning_fabric/plugins/environments/slurm.py around the check reported at line 204. Reproduce the salloc followed by srun command from the report and compare it with srun -n1 --pty bash; done means the valid interactive allocation no longer triggers the warning while the invalid case is still detected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.