Lightning-AI / Lightning-AI/pytorch-lightning

Restoring Trainer State with Early Stop fails

Open
#13,225 9 comments 8 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug callback: early stopping repro needed waiting on author won't fix
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🐛 Bug

Whenever I try to restore the state of a previous (broken) run and I have Early Stop configured, I got the following error:

```
RuntimeError: Early stopping conditioned on metric `val/f1/default` which is not available. Pass in or modify your `EarlyStopping` callback to use any of the following: ``
```

It seems the metrics are not saved together with the Checkpoint data and Early Stop tries to be applied straight away.

### To Reproduce

1. Start training a model with Checkpointing (best and last) and Early Stop on a validation metric. I have used Pytorch-Lightning Cli for this.
2. The model crashes after some checkpoints were saved
3. Try to re-run the training passing `ckpt_path` with the last checkpoint path to the trainer (with the same Early Stopping settings).
4. The error happens.

### Expected behavior

The model should resume training, knowing the last best value, and apply Early Stopping as appropriate.

### Environment

* CUDA:
- GPU:
- NVIDIA A100-SXM4-80GB
- available: True
- version: 11.4
* Packages:
- numpy: 1.21.2
- pyTorch_debug: False
- pyTorch_version: 1.10.0a0+0aef44c
- pytorch-lightning: 1.6.1
- tqdm: 4.62.3
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.8.12
- version: #103-Ubuntu SMP Fri Nov 26 16:13:00 UTC 2021

### Additional context

Running using the Pytorch Lightning Cli

cc @carmocca @awaelchli @rohitgr7

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with checkpoint restoration, Early Stopping, and a validation metric such as `val/f1/default`, then inspect how checkpoint state and EarlyStopping state are restored. Confirm the fix by resuming from the last checkpoint after a crash and verifying that training continues and Early Stopping uses the restored best value.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.