Lightning-AI / Lightning-AI/pytorch-lightning

Global step reset when restoring checkpoints with trainer.validate

Open
#17,127 8 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

checkpointing feature loops pl
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

When restoring checkpoints with trainer.validate, global_step and epoch are overwritten with 0.

It should keep the same global_step and epoch otherwise, it messes with the loggers.

This issue prevents to correctly validate checkpoints of a model as a postprocessing.

### How to reproduce the bug

run a model with
```python

trainer.validate(model, datamodule, cpt_path=ckpt_path

```

and log a metric, the result will be logged at step 0.

### Environment

Current environment

```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0): 1.9
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0): 1.13
#- Python version (e.g., 3.9): 3.10
#- OS (e.g., Linux): Linux
#- CUDA/cuDNN version: 11.7
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
```

### More info

_No response_

cc @borda @awaelchli @carmocca @justusschock

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the trainer.validate checkpoint-restoration path and reproduce the issue with the provided Python call, logging the resulting step and epoch. Trace where checkpoint state is restored during validation; done means validation preserves the checkpoint's global_step and epoch so logged metrics use the restored step.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.