Lightning-AI / Lightning-AI/pytorch-lightning
trainer.fit from checkpoint without performance improvement will break 'last' link to checkpoint on window11
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
just as titled, training a model on window11, pass a checkpoint callback to trainer and keep ckpt_path as None as code below, then fit model with data and lightning will create link well to checkpoint file.
then trains the same model again but load model from ckpt_path, this time make it no improvement while fitting model. after training done then 'last' link become wrong.
### What version are you seeing the problem on?
v2.2
### How to reproduce the bug
```python
checkpoint_callback = ModelCheckpoint(
monitor='val_loss', # 监控的指标
dirpath='training/checkpoints/', # 保存检查点的目录
filename=experiment_name+'-{epoch}-{val_loss:.3f}', # 检查点文件名的格式
save_top_k=1, # 仅保存最佳的一个模型
mode='min', # 因为是损失,所以越小越好
save_last='link',
save_on_train_epoch_end=True,
every_n_epochs=5
)
...
trainer.fit(model, ckpt_path=None if initial else 'last')
trainer.test(model)
```
### Error messages and logs
```
# Error messages and logs here please
```
### Environment
Current environment
```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
```
### More info
_No response_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the Windows 11 scenario using ModelCheckpoint and trainer.fit: first create a checkpoint with save_last='link', then resume from 'last' without metric improvement. Start by tracing the ModelCheckpoint handling of save_last and checkpoint links, and verify that the 'last' link still points to the correct checkpoint after the resumed fit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100