Lightning-AI / Lightning-AI/pytorch-lightning
`ModelCheckpoint` not saving best model
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
In DDP, the `ModelCheckpooint` (configuration below) does not save the best model despite lower validation losses being achieved in later epochs. Configuration:
```
checkpoint_callback = ModelCheckpoint(
dirpath=path_save_model,
filename="best_loss",
monitor="val_loss",
mode="min",
every_n_epochs=1,
verbose=True,
)
```
Here is a snippet of the output log:
```
Epoch 292: 100% 40/40 [00:49<00:00, 1.24s/it, v_num=0, l1=0.00589, msssim=0.00441, lpips=0.0184, loss=0.0287, lr=0.000128, val_l1=0.00746, val_msssim=0.0122, val_lpips=0.0299, val_loss=0.0495]Epoch 292, global step 11720: 'val_loss' reached 0.04955 (best 0.04955), saving model to '/qumulo/sravan/Projects/pmc_lit/experiments/e3_uwsyn_multi_noprompt/250318_2110_30k_uw50pc_ghosting/best_loss.ckpt' as top 1
Epoch 297: 100% 40/40 [00:49<00:00, 1.24s/it, v_num=0, l1=0.0141, msssim=0.00869, lpips=0.0113, loss=0.0341, lr=0.000124, val_l1=0.00779, val_msssim=0.0121, val_lpips=0.0266, val_loss=0.0466]Epoch 297, global step 11920: 'val_loss' was not in top 1
```
Clearly, epoch 297 should be saved to disk
### What version are you seeing the problem on?
v2.5
### How to reproduce the bug
```python
```
### Error messages and logs
```
Epoch 292: 100% 40/40 [00:49<00:00, 1.24s/it, v_num=0, l1=0.00589, msssim=0.00441, lpips=0.0184, loss=0.0287, lr=0.000128, val_l1=0.00746, val_msssim=0.0122, val_lpips=0.0299, val_loss=0.0495]Epoch 292, global step 11720: 'val_loss' reached 0.04955 (best 0.04955), saving model to '250318_2110_30k_uw50pc/best_loss.ckpt' as top 1
Epoch 297: 100% 40/40 [00:49<00:00, 1.24s/it, v_num=0, l1=0.0141, msssim=0.00869, lpips=0.0113, loss=0.0341, lr=0.000124, val_l1=0.00779, val_msssim=0.0121, val_lpips=0.0266, val_loss=0.0466]Epoch 297, global step 11920: 'val_loss' was not in top 1
```
### Environment
Current environment
```
#- PyTorch Lightning Version (e.g., 2.5.0): 2.5.0
#- PyTorch Version (e.g., 2.5): 2.3.1+cu121
#- Python version (e.g., 3.12): 3.10.12
#- OS (e.g., Linux): Ubuntu 22.04.3 LTS
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source): pip
```
### More info
_No response_
cc @justusschock
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the ModelCheckpoint behavior for top-1 monitoring in DDP on v2.5, using the supplied configuration and log entries as the failing case. Verify how the val_loss values from epochs 292 and 297 are compared and persisted; done means the later lower value is retained as the best checkpoint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100