Lightning-AI / Lightning-AI/pytorch-lightning

OnExceptionCheckpoint: training resumes if ckpt found, even if no ckpt_path provided

Open
#19,827 0 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug checkpointing trainer: fit
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

- The [documentation of `Trainer.fit`](https://lightning.ai/docs/pytorch/stable/common/trainer.html#fit) states training will only be resumed if `ckpt_path` is provided.
- The [documenation of `OnExceptionCheckpoint`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.OnExceptionCheckpoint.html#onexceptioncheckpoint) states it's purpose is to save a checkpoint on exception.

However, if an `OnExceptionCheckpoint` is provided to the trainer's list of callbacks, then even if `Trainer.fit` is called without providing `ckpt_path` argument then the `CheckpointConnector` will search for a checkpoint in `OnExceptionCheckpoint.dirpath`, and if one is found it will be used to resume training.

Further, the warnings shown at the beginning of `Trainer.fit` when `OnExceptionCheckpoint` callback is enabled IMO are incorrect, (see logs below)

### What version are you seeing the problem on?

v2.2

### How to reproduce the bug

```python
from lightning.pytorch.callbacks import OnExceptionCheckpoint, ModelCheckpoint
from lightning.pytorch import Trainer
from lightning.pytorch.demos.boring_classes import BoringModel

trainer = Trainer(
# NOTE: either of these callback lists reproduce the issue, (and same warnings mentioned below)
# callbacks=[OnExceptionCheckpoint("."), ModelCheckpoint(".", save_last=True)],
callbacks=[OnExceptionCheckpoint("."), ModelCheckpoint(".", save_last=False)],
max_epochs=3,
)
trainer.fit(model=BoringModel())

# calling `fit` again will result in resumed training from discovered checkpoint in cwd:
trainer.fit(model=BoringModel())
```

### Error messages and logs

The following warnings are confusing and arguably wrong:

```
.../checkpoint_connector.py:126: `.fit(ckpt_path=None)` was called without a model. The last model of the previous `fit` call will be used. You can pass `fit(ckpt_path='best')` to use the best model or `fit(ckpt_path='last')` to use the last model. If you pass a value, this warning will be silenced.
.../checkpoint_connector.py:186: .fit(ckpt_path="last") is set, but there is no last checkpoint available. No checkpoint will be loaded. HINT: Set `ModelCheckpoint(..., save_last=True)`.```

### Environment

```
PyTorch Lightning Version: 2.2.1
```

### More info

_No response_

cc @lantiga

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Trainer.fit and the CheckpointConnector warnings referenced at checkpoint_connector.py lines 126 and 186, then inspect how OnExceptionCheckpoint.dirpath is handled during checkpoint discovery. Reproduce the two consecutive fit calls from the issue and add or update tests so a fit without ckpt_path does not resume from an exception checkpoint and the warnings match that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.