Lightning-AI / Lightning-AI/pytorch-lightning

trainer.fit from checkpoint without performance improvement will break 'last' link to checkpoint on window11

Open
#19,845 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug callback: model checkpoint
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

just as titled, training a model on window11, pass a checkpoint callback to trainer and keep ckpt_path as None as code below, then fit model with data and lightning will create link well to checkpoint file.
then trains the same model again but load model from ckpt_path, this time make it no improvement while fitting model. after training done then 'last' link become wrong.

### What version are you seeing the problem on?

v2.2

### How to reproduce the bug

```python
checkpoint_callback = ModelCheckpoint(
monitor='val_loss', # 监控的指标
dirpath='training/checkpoints/', # 保存检查点的目录
filename=experiment_name+'-{epoch}-{val_loss:.3f}', # 检查点文件名的格式
save_top_k=1, # 仅保存最佳的一个模型
mode='min', # 因为是损失,所以越小越好
save_last='link',
save_on_train_epoch_end=True,
every_n_epochs=5
)
...
trainer.fit(model, ckpt_path=None if initial else 'last')
trainer.test(model)
```

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
```

### More info

_No response_

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the Windows 11 scenario using ModelCheckpoint and trainer.fit: first create a checkpoint with save_last='link', then resume from 'last' without metric improvement. Start by tracing the ModelCheckpoint handling of save_last and checkpoint links, and verify that the 'last' link still points to the correct checkpoint after the resumed fit.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.