Lightning-AI / Lightning-AI/pytorch-lightning
Turn off hpc checkpoint saving in SLURM environment if trainer.fit(..., ckpt_path="last")
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
Hi all,
`trainer.fit(ckpt_path="last")` works perfectly now (in SLURM environment with autorequeue=True), especially if you set up end-of-epoch checkpoint callback. However, the trainer still saves middle-of-the-epoch checkpoints that remain unused because of `trainer.fit(ckpt_path="last")`. It would be great to not save hpc_ckpt if trainer.fit(ckpt_path="last").
This feature will close #13773
### Pitch
_No response_
### Alternatives
_No response_
### Additional context
_No response_
cc @lantiga @borda
Contributor guide
Research direction
Start at the trainer.fit(..., ckpt_path="last") entry point and trace how SLURM autorequeue mode handles middle-of-epoch HPC checkpoints. Determine where the checkpoint-saving decision is made; done means hpc_ckpt is not saved for this mode while existing resume behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- hpc
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100