Lightning-AI / Lightning-AI/pytorch-lightning
Checkpoint callback run before validation step - stale or none monitor values considered for validation metrics
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
I am doing iterative training with `check_val_every_n_epoch=None` and (example values)`val_check_interval=10` on my trainer and with the matched argument on ModelCheckpoint `every_n_train_steps=10`.
e.g.
```python
checkpoint_callback = ModelCheckpoint(
dirpath=experiment_dir.joinpath("checkpoints"),
filename="checkpoint-{epoch}-{step:06d}-{train_loss:.2f}-{val_loss:.2f}",
save_top_k=checkpoint_top_k,
every_n_train_steps=checkpoint_n_step,
monitor="val_loss",
)
```
It is a [documented](https://lightning.ai/docs/pytorch/stable/common/checkpointing_intermediate.html) usage to make the monitor metric `val_loss`.
The problem is that these values might not exist, giving the [warning](https://github.com/Lightning-AI/pytorch-lightning/blob/1551a16b94f5234a4a78801098f64d0732ef5cb5/src/lightning/pytorch/callbacks/model_checkpoint.py#L378) or they are stale - because val_step is run after the checkpoint has been processed, new val metrics are not considered.
### What version are you seeing the problem on?
v2.3, v2.4
### How to reproduce the bug
_No response_
### Error messages and logs
```
# Error messages and logs here please
```
### Environment
Current environment
```
#- PyTorch Lightning Version (e.g., 2.4.0):
#- PyTorch Version (e.g., 2.4):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
```
### More info
_No response_
Contributor guide
Research direction
Start in src/lightning/pytorch/callbacks/model_checkpoint.py around the warning referenced in the issue, then trace the ordering of validation and checkpoint callbacks for val_check_interval and every_n_train_steps. Reproduce the shown configuration with check_val_every_n_epoch=None and verify that checkpoint monitoring uses the latest validation metrics rather than missing or stale values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100