Checkpointing in signal handlers for SLURM auto-requeueing leads to intermittent failures

Nobody has claimed this yet.

### Bug description

I have been experienced intermittent and difficult to pin down failures when lightning tried to auto-requeue my jobs on our SLURM cluster on time out. After some debugging (`print` in the signal handler for `SIGUSR1` and many runs), I saw that sometimes the handler would just stop running after or while saving the HPC checkpoint to disk.

Signal handlers are a pretty special environment, because they can run after any python bytecode instruction, so also in the middle of other functions. This can lead to difficult to track down problems and also crashes, e.g. https://github.com/python/cpython/issues/112608. Or it could run in the middle of a backward pass and store a corrupted checkpoint. So the safe option would be to do as little work as possible in them and move the actual signal handling into normal code.

### What version are you seeing the problem on?

v2.5, master

### Reproduced in studio

_No response_

### How to reproduce the bug

The issue is difficult to reproduce as it depends on signal timing and the exact environment. It never happens on one cluster but regularly on the other.

### Error messages and logs

No error messages. The signal handler just never completes until the program gets killed by SLURM.

### Environment

Current environment

* CUDA:
- GPU: None
- available: False
- version: 12.6
* Lightning:
- lightning: 2.6.0
- lightning-utilities: 0.15.2
- pytorch-lightning: 2.6.0
- torch: 2.9.1+cu126
- torch-fidelity: 0.3.0
- torchdata: 0.11.0
- torchmetrics: 1.8.2
- torchvision: 0.24.1+cu126
* Packages:
- aiohappyeyeballs: 2.6.1
- aiohttp: 3.13.2
- aiosignal: 1.4.0
- annotated-types: 0.7.0
- antlr4-python3-runtime: 4.9.3
- anyio: 4.12.0
- argon2-cffi: 25.1.0
- argon2-cffi-bindings: 25.1.0
- arrow: 1.4.0
- asttokens: 3.0.1
- async-lru: 2.0.5
- attrs: 25.4.0
- autocommand: 2.2.2
- babel: 2.17.0
- backports.tarfile: 1.2.0
- beautifulsoup4: 4.14.3
- bleach: 6.3.0
- brezn: 0.1.0
- bsi: 0.1.0
- cachetools: 6.2.2
- cattrs: 25.3.0
- certifi: 2025.11.12
- cffi: 2.0.0
- charset-normalizer: 3.4.4
- click: 8.3.1
- cloudpickle: 3.1.2
- comm: 0.2.3
- contourpy: 1.3.3
- cycler: 0.12.1
- debugpy: 1.8.17
- decorator: 5.2.1
- defusedxml: 0.7.1
- einops: 0.8.1
- executing: 2.2.1
- fastjsonschema: 2.21.2
- filelock: 3.20.0
- fonttools: 4.61.0
- fqdn: 1.5.1
- frozenlist: 1.8.0
- fsspec: 2025.12.0
- gitdb: 4.0.12
- gitignorant: 0.4.0
- gitpython: 3.1.45
- h11: 0.16.0
- h5py: 3.15.1
- httpcore: 1.0.9
- httpx: 0.28.1
- hydra-core: 1.3.2
- hydra-submitit-launcher: 1.4.0.dev0
- idna: 3.11
- importlib-metadata: 8.0.0
- inflect: 7.3.1
- iniconfig: 2.3.0
- ipdb: 0.13.13
- ipykernel: 7.1.0
- ipympl: 0.9.8
- ipython: 9.8.0
- ipython-pygments-lexers: 1.1.1
- ipywidgets: 8.1.8
- isoduration: 20.11.0
- jaraco.collections: 5.1.0
- jaraco.context: 5.3.0
- jaraco.functools: 4.0.1
- jaraco.text: 3.12.1
- jaxtyping: 0.3.3
- jedi: 0.19.2
- jinja2: 3.1.6
- json5: 0.12.1
- jsonpointer: 3.0.0
- jsonschema: 4.25.1
- jsonschema-specifications: 2025.9.1
- jupyter-client: 8.6.3
- jupyter-core: 5.9.1
- jupyter-events: 0.12.0
- jupyter-lsp: 2.3.0
- jupyter-server: 2.17.0
- jupyter-server-terminals: 0.5.3
- jupyterlab: 4.5.0
- jupyterlab-pygments: 0.3.0
- jupyterlab-server: 2.28.0
- jupyterlab-widgets: 3.0.16
- kiwisolver: 1.4.9
- lark: 1.3.1
- lightning: 2.6.0
- lightning-utilities: 0.15.2
- loky: 3.5.6
- markdown-it-py: 4.0.0
- markupsafe: 3.0.3
- matplotlib: 3.10.7
- matplotlib-inline: 0.2.1
- mdurl: 0.1.2
- mistune: 3.1.4
- more-itertools: 10.3.0
- mpmath: 1.3.0
- multidict: 6.7.0
- nbclient: 0.10.2
- nbconvert: 7.16.6
- nbformat: 5.10.4
- nest-asyncio: 1.6.0
- networkx: 3.6
- notebook-shim: 0.2.4
- numpy: 2.3.5
- nvidia-cublas-cu12: 12.6.4.1
- nvidia-cuda-cupti-cu12: 12.6.80
- nvidia-cuda-nvrtc-cu12: 12.6.77
- nvidia-cuda-runtime-cu12: 12.6.77
- nvidia-cudnn-cu12: 9.10.2.21
- nvidia-cufft-cu12: 11.3.0.4
- nvidia-cufile-cu12: 1.11.1.6
- nvidia-curand-cu12: 10.3.7.77
- nvidia-cusolver-cu12: 11.7.1.2
- nvidia-cusparse-cu12: 12.5.4.2
- nvidia-cusparselt-cu12: 0.7.1
- nvidia-nccl-cu12: 2.27.5
- nvidia-nvjitlink-cu12: 12.6.85
- nvidia-nvshmem-cu12: 3.3.20
- nvidia-nvtx-cu12: 12.6.77
- omegaconf: 2.3.0
- packaging: 25.0
- pandocfilters: 1.5.1
- parso: 0.8.5
- pexpect: 4.9.0
- pillow: 12.0.0
- platformdirs: 4.5.0
- pluggy: 1.6.0
- prometheus-client: 0.23.1
- prompt-toolkit: 3.0.52
- propcache: 0.4.1
- protobuf: 6.33.1
- psutil: 7.1.3
- ptyprocess: 0.7.0
- pure-eval: 0.2.3
- pycparser: 2.23
- pydantic: 2.12.5
- pydantic-core: 2.41.5
- pygments: 2.19.2
- pyparsing: 3.2.5
- pytest: 9.0.1
- python-dateutil: 2.9.0.post0
- python-json-logger: 4.0.0
- pytorch-lightning: 2.6.0
- pyyaml: 6.0.3
- pyzmq: 27.1.0
- referencing: 0.37.0
- requests: 2.32.5
- rfc3339-validator: 0.1.4
- rfc3986-validator: 0.1.1
- rfc3987-syntax: 1.1.0
- rich: 14.2.0
- rpds-py: 0.30.0
- scipy: 1.16.3
- send2trash: 1.8.3
- sentry-sdk: 2.47.0
- setuptools: 80.9.0
- six: 1.17.0
- smmap: 5.0.2
- soupsieve: 2.8
- stack-data: 0.6.3
- submitit: 1.5.3
- sympy: 1.14.0
- terminado: 0.18.1
- tinycss2: 1.4.0
- toml: 0.10.2
- tomli: 2.0.1
- torch: 2.9.1+cu126
- torch-fidelity: 0.3.0
- torchdata: 0.11.0
- torchmetrics: 1.8.2
- torchvision: 0.24.1+cu126
- tornado: 6.5.2
- tqdm: 4.67.1
- traitlets: 5.14.3
- triton: 3.5.1
- typeguard: 4.3.0
- typing-extensions: 4.15.0
- typing-inspection: 0.4.2
- tzdata: 2025.2
- uri-template: 1.3.0
- urllib3: 2.5.0
- wadler-lindig: 0.1.7
- wandb: 0.23.1
- wcwidth: 0.2.14
- webcolors: 25.10.0
- webencodings: 0.5.1
- websocket-client: 1.9.0
- wheel: 0.45.1
- widgetsnbextension: 4.0.15
- yarl: 1.22.0
- zipp: 3.19.2
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.13.2
- release: 5.15.0-161-generic
- version: #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025

### More info

_No response_

cc @ethanwharris

The issue names no source files or tests. Trace the SLURM SIGUSR1 handler and checkpoint-saving path first, then determine how the signal work can be deferred to normal execution; done means auto-requeueing no longer leaves checkpoint handling stuck or vulnerable to interruption during a save.

Written by the indexing model from the issue text.