Lightning-AI / Lightning-AI/pytorch-lightning

Checkpoint callback run before validation step - stale or none monitor values considered for validation metrics

Open
#20,185 2 comments 0 reactions 0 assignees View on GitHub
bug callback: model checkpoint ver: 2.3.x ver: 2.4.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

I am doing iterative training with `check_val_every_n_epoch=None` and (example values)`val_check_interval=10` on my trainer and with the matched argument on ModelCheckpoint `every_n_train_steps=10`.

e.g.
```python
checkpoint_callback = ModelCheckpoint(
dirpath=experiment_dir.joinpath("checkpoints"),
filename="checkpoint-{epoch}-{step:06d}-{train_loss:.2f}-{val_loss:.2f}",
save_top_k=checkpoint_top_k,
every_n_train_steps=checkpoint_n_step,
monitor="val_loss",
)
```

It is a [documented](https://lightning.ai/docs/pytorch/stable/common/checkpointing_intermediate.html) usage to make the monitor metric `val_loss`.

The problem is that these values might not exist, giving the [warning](https://github.com/Lightning-AI/pytorch-lightning/blob/1551a16b94f5234a4a78801098f64d0732ef5cb5/src/lightning/pytorch/callbacks/model_checkpoint.py#L378) or they are stale - because val_step is run after the checkpoint has been processed, new val metrics are not considered.

### What version are you seeing the problem on?

v2.3, v2.4

### How to reproduce the bug

_No response_

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

```
#- PyTorch Lightning Version (e.g., 2.4.0):
#- PyTorch Version (e.g., 2.4):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
```

### More info

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in src/lightning/pytorch/callbacks/model_checkpoint.py around the warning referenced in the issue, then trace the ordering of validation and checkpoint callbacks for val_check_interval and every_n_train_steps. Reproduce the shown configuration with check_val_every_n_epoch=None and verify that checkpoint monitoring uses the latest validation metrics rather than missing or stale values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.