Lightning-AI / Lightning-AI/pytorch-lightning
OnExceptionCheckpoint: training resumes if ckpt found, even if no ckpt_path provided
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
- The [documentation of `Trainer.fit`](https://lightning.ai/docs/pytorch/stable/common/trainer.html#fit) states training will only be resumed if `ckpt_path` is provided.
- The [documenation of `OnExceptionCheckpoint`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.OnExceptionCheckpoint.html#onexceptioncheckpoint) states it's purpose is to save a checkpoint on exception.
However, if an `OnExceptionCheckpoint` is provided to the trainer's list of callbacks, then even if `Trainer.fit` is called without providing `ckpt_path` argument then the `CheckpointConnector` will search for a checkpoint in `OnExceptionCheckpoint.dirpath`, and if one is found it will be used to resume training.
Further, the warnings shown at the beginning of `Trainer.fit` when `OnExceptionCheckpoint` callback is enabled IMO are incorrect, (see logs below)
### What version are you seeing the problem on?
v2.2
### How to reproduce the bug
```python
from lightning.pytorch.callbacks import OnExceptionCheckpoint, ModelCheckpoint
from lightning.pytorch import Trainer
from lightning.pytorch.demos.boring_classes import BoringModel
trainer = Trainer(
# NOTE: either of these callback lists reproduce the issue, (and same warnings mentioned below)
# callbacks=[OnExceptionCheckpoint("."), ModelCheckpoint(".", save_last=True)],
callbacks=[OnExceptionCheckpoint("."), ModelCheckpoint(".", save_last=False)],
max_epochs=3,
)
trainer.fit(model=BoringModel())
# calling `fit` again will result in resumed training from discovered checkpoint in cwd:
trainer.fit(model=BoringModel())
```
### Error messages and logs
The following warnings are confusing and arguably wrong:
```
.../checkpoint_connector.py:126: `.fit(ckpt_path=None)` was called without a model. The last model of the previous `fit` call will be used. You can pass `fit(ckpt_path='best')` to use the best model or `fit(ckpt_path='last')` to use the last model. If you pass a value, this warning will be silenced.
.../checkpoint_connector.py:186: .fit(ckpt_path="last") is set, but there is no last checkpoint available. No checkpoint will be loaded. HINT: Set `ModelCheckpoint(..., save_last=True)`.```
### Environment
```
PyTorch Lightning Version: 2.2.1
```
### More info
_No response_
cc @lantiga
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Trainer.fit and the CheckpointConnector warnings referenced at checkpoint_connector.py lines 126 and 186, then inspect how OnExceptionCheckpoint.dirpath is handled during checkpoint discovery. Reproduce the two consecutive fit calls from the issue and add or update tests so a fit without ckpt_path does not resume from an exception checkpoint and the warnings match that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100