Epoch Skipping During Training After Multiple Jobs Submitted On Same Slurm Node

Nobody has claimed this yet.

### Bug description

This has me stumped, can't finish an epoch without skipping. Submitting multiple jobs on a slurm cluster using one V100 GPU. The GPU nodes have 4 V100s each. Jobs can run on the same node, wondering if that could be the reason for the skipping? Can anyone suggest a way to debug this?

### What version are you seeing the problem on?

v2.0

### How to reproduce the bug

```python
# this is the setup in my trainer.py

checkpoint_callback = ModelCheckpoint( monitor="val_loss",
filename=str(args.model + '-epoch{epoch:02d}-val_loss{val/loss:.2f}'),
auto_insert_metric_name=False,
mode="min",
save_last=True,
save_top_k=3,)

accumulator = GradientAccumulationScheduler(scheduling={1000: 2, 2000:4})

trainer = pl.Trainer(
accelerator="gpu",
devices=[0], # set to -1 to use all avaliable gpus...
# strategy='ddp', # should be same as args.backend..., # stochastic_weight_avg=True, # pass to callbacks if required...
reload_dataloaders_every_n_epochs=1,
# limit_train_batches=0.2,
# limit_train_batches=0.7,
# limit_val_batches=0.5,
default_root_dir=model.hparams.root,
max_epochs=model.hparams.n_epochs*2,
check_val_every_n_epoch=1,
# log_gpu_memory='min_max',
sync_batchnorm=True,
log_every_n_steps=10,
precision="16",
callbacks=[checkpoint_callback, accumulator],
deterministic=True)
# log_every_n_steps=30)

trainer.fit(model)
```

### Error messages and logs

```
# This is what it prints, doesn't go through validation

Epoch 0: 8%|▊ | 41/522 [11:12<2:11:35, 16.41s/it, v_num=4, train_loss_step=8.190]tensor(25, device='cuda:0')
Epoch 1: 0%| | 0/522 [00:00
* CUDA:
- GPU: 1
- available: True
- version: 11.7
* Lightning:
- lightning: 2.0.6
- lightning-cloud: 0.5.37
- lightning-utilities: 0.9.0
- pytorch-lightning: 2.0.5
- torch: 2.0.1
- torchmetrics: 1.0.0
* Packages:
- acvl-utils: 0.2
- aiohttp: 3.8.4
- aiosignal: 1.3.1
- anyio: 3.7.1
- argparse: 1.4.0
- arrow: 1.2.3
- async-timeout: 4.0.2
- attrs: 23.1.0
- backoff: 2.2.1
- batchgenerators: 0.25
- beautifulsoup4: 4.12.2
- blessed: 1.20.0
- certifi: 2023.5.7
- charset-normalizer: 3.1.0
- click: 8.1.4
- cmake: 3.26.3
- connected-components-3d: 3.10.5
- contourpy: 1.0.7
- croniter: 1.4.1
- cycler: 0.11.0
- dateutils: 0.6.12
- deepdiff: 6.3.1
- dicom2nifti: 2.4.8
- dynamic-network-architectures: 0.2
- elasticdeform: 0.5.0
- fastapi: 0.100.0
- filelock: 3.12.0
- fonttools: 4.39.4
- frozenlist: 1.3.3
- fsspec: 2023.6.0
- future: 0.18.3
- graphviz: 0.20.1
- h11: 0.14.0
- h5py: 3.9.0
- hiddenlayer: 0.2
- idna: 3.4
- imagecodecs: 2023.3.16
- imageio: 2.29.0
- inquirer: 3.1.3
- itsdangerous: 2.1.2
- jinja2: 3.1.2
- joblib: 1.2.0
- kiwisolver: 1.4.4
- lazy-loader: 0.2
- lightning: 2.0.6
- lightning-cloud: 0.5.37
- lightning-utilities: 0.9.0
- linecache2: 1.0.0
- lit: 16.0.5
- markdown-it-py: 3.0.0
- markupsafe: 2.1.2
- matplotlib: 3.7.1
- mdurl: 0.1.2
- medpy: 0.4.0
- monai: 1.2.0
- mpmath: 1.3.0
- multidict: 6.0.4
- networkx: 3.1
- nibabel: 5.1.0
- nnunetv2: 2.1
- nptyping: 2.5.0
- numpy: 1.24.3
- nvidia-cublas-cu11: 11.10.3.66
- nvidia-cuda-cupti-cu11: 11.7.101
- nvidia-cuda-nvrtc-cu11: 11.7.99
- nvidia-cuda-runtime-cu11: 11.7.99
- nvidia-cudnn-cu11: 8.5.0.96
- nvidia-cufft-cu11: 10.9.0.58
- nvidia-curand-cu11: 10.2.10.91
- nvidia-cusolver-cu11: 11.4.0.1
- nvidia-cusparse-cu11: 11.7.4.91
- nvidia-nccl-cu11: 2.14.3
- nvidia-nvtx-cu11: 11.7.91
- opencv-python: 4.8.0.74
- ordered-set: 4.1.0
- packaging: 23.1
- pandas: 2.0.1
- pillow: 9.5.0
- pip: 23.1.2
- psutil: 5.9.5
- pydantic: 1.10.11
- pydicom: 2.3.1
- pygments: 2.15.1
- pyjwt: 2.7.0
- pynrrd: 1.0.0
- pyparsing: 3.0.9
- python-dateutil: 2.8.2
- python-editor: 1.0.4
- python-gdcm: 3.0.22
- python-multipart: 0.0.6
- pytorch-lightning: 2.0.5
- pytz: 2023.3
- pywavelets: 1.4.1
- pyyaml: 6.0
- readchar: 4.0.5
- requests: 2.31.0
- rich: 13.4.2
- scikit-image: 0.20.0
- scikit-learn: 1.2.2
- scipy: 1.10.1
- seaborn: 0.12.2
- setuptools: 67.7.2
- simpleitk: 2.2.1
- six: 1.16.0
- sniffio: 1.3.0
- soupsieve: 2.4.1
- starlette: 0.27.0
- starsessions: 1.3.0
- sympy: 1.12
- threadpoolctl: 3.1.0
- tifffile: 2023.4.12
- torch: 2.0.1
- torchmetrics: 1.0.0
- tqdm: 4.65.0
- traceback2: 1.4.0
- traitlets: 5.9.0
- triton: 2.0.0
- typing-extensions: 4.6.0
- tzdata: 2023.3
- unittest2: 1.1.0
- urllib3: 2.0.2
- uvicorn: 0.22.0
- wcwidth: 0.2.6
- websocket-client: 1.6.1
- websockets: 11.0.3
- wheel: 0.40.0
- yacs: 0.1.8
- yarl: 1.9.2
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.11.3
- release: 3.10.0-1160.36.2.el7.x86_64
- version: #1 SMP Wed Jul 21 11:57:15 UTC 2021

```

### More info

_No response_

cc @justusschock @awaelchli

The reproduction is in trainer.py and uses ModelCheckpoint, GradientAccumulationScheduler, and a single GPU under Slurm. Start by reproducing the epoch transition with one job and with multiple jobs on the same node, then inspect the printed batch and epoch values. Done means identifying whether the skipped batches come from the training setup or the framework and documenting or fixing the confirmed cause.

Written by the indexing model from the issue text.