Lightning-AI / Lightning-AI/pytorch-lightning
Mid-epoch resume causes a single unwanted validation step (which is not a sanity check)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
This is an issue created from the discussion in this thread: https://github.com/Lightning-AI/pytorch-lightning/discussions/18110#discussioncomment-10115118 which seems to affect a few people.
When resuming from a checkpoint using mid-epoch checkpointing (in my case I use the ModelCheckpoint callback with `train_time_interval` every 1 hour), two primary cases arise:
- The checkpoint was in mid-epoch and the training resumes without issues.
- The checkpoint was at end of epoch, just before validation, in which case the model loads and performs a single validation step which causes an evaluation with a single batch, and alters metrics at this point.
Certain relevant elements:
- Validation is only performed once every epoch (at the end)
- I use two checkpoint callbacks: one with time interval and one at the end of epoch during training.
- This seems to happen when an error occurs during validation.
Do you know if this issue can be solved or bypassed ? What are the reasons this happens ?
### What version are you seeing the problem on?
v2.2
### How to reproduce the bug
_No response_
### Error messages and logs
```
# Error messages and logs here please
```
### Environment
Current environment
```
- PyTorch Lightning Version: 2.2.0
- PyTorch Version: 2.0.0
- Python version: 3.11.8
```
### More info
Don't have a minimal working example, but if need be I can try to make one.
cc @awaelchli
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the ModelCheckpoint callback and resume flow using train_time_interval alongside end-of-epoch validation, reproducing the reported v2.2 setup if possible. A complete fix should resume from an end-of-epoch checkpoint without performing an unwanted single-batch validation or altering metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100