Why does running Lightning on SLURM with python perform worse than with srun?

Nobody has claimed this yet.

### Bug description

I'm training LLMs across multiple GPUs on a single node using `Nvidia/NeMo`.
When launching via `python train.py` inside of an allocation I get much worse performance than when launching directly via `srun`. In the first case Pytorch Lightning also raises the warning: `The srun command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with srun`. See below:

107 tflops:
```
srun --unbuffered --exclusive --open-mode=append --partition=ultra --nodes=1 --ntasks-per-node=4 --gpus=4 --pty bash
source /home/niklas/miniconda/bin/activate
conda activate core
CUDA_DEVICE_MAX_CONNECTIONS=1 GPU_PER_NODE=4 python train.py
```

172 tflops:
```
source /home/niklas/miniconda/bin/activate
conda activate core
CUDA_DEVICE_MAX_CONNECTIONS=1 GPU_PER_NODE=4 srun --unbuffered --exclusive --open-mode=append --partition=ultra --nodes=1 --ntasks-per-node=4 --gpus=4 python -u train.py
```

Why is it the case that the first one performs worse? Maybe is there a difference in how these two strategies launch the torch.distributed process group? (https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing)

### What version are you seeing the problem on?

v2.0

### How to reproduce the bug

_No response_

### Error messages and logs

_No response_

### Environment

Current environment

```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
```

### More info

_No response_

cc @awaelchli

Start by comparing the two provided SLURM launch commands and the Lightning warning, then inspect the SLURM integration and distributed process launch behavior around running train.py. Reproduce the performance difference with the supplied commands and complete the investigation when the cause is explained and the two launch modes behave consistently or the documented distinction is clear.

Written by the indexing model from the issue text.