Lightning-AI / Lightning-AI/pytorch-lightning

Mid-epoch resume causes a single unwanted validation step (which is not a sanity check)

Open
#20,288 4 comments 11 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug repro needed reproducibility
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

This is an issue created from the discussion in this thread: https://github.com/Lightning-AI/pytorch-lightning/discussions/18110#discussioncomment-10115118 which seems to affect a few people.

When resuming from a checkpoint using mid-epoch checkpointing (in my case I use the ModelCheckpoint callback with `train_time_interval` every 1 hour), two primary cases arise:
- The checkpoint was in mid-epoch and the training resumes without issues.
- The checkpoint was at end of epoch, just before validation, in which case the model loads and performs a single validation step which causes an evaluation with a single batch, and alters metrics at this point.

Certain relevant elements:
- Validation is only performed once every epoch (at the end)
- I use two checkpoint callbacks: one with time interval and one at the end of epoch during training.
- This seems to happen when an error occurs during validation.

Do you know if this issue can be solved or bypassed ? What are the reasons this happens ?

### What version are you seeing the problem on?

v2.2

### How to reproduce the bug

_No response_

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

```
- PyTorch Lightning Version: 2.2.0
- PyTorch Version: 2.0.0
- Python version: 3.11.8
```

### More info

Don't have a minimal working example, but if need be I can try to make one.

cc @awaelchli

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ModelCheckpoint callback and resume flow using train_time_interval alongside end-of-epoch validation, reproducing the reported v2.2 setup if possible. A complete fix should resume from an end-of-epoch checkpoint without performing an unwanted single-batch validation or altering metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.