Lightning-AI / Lightning-AI/pytorch-lightning
Restoring Trainer State with Early Stop fails
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🐛 Bug
Whenever I try to restore the state of a previous (broken) run and I have Early Stop configured, I got the following error:
```
RuntimeError: Early stopping conditioned on metric `val/f1/default` which is not available. Pass in or modify your `EarlyStopping` callback to use any of the following: ``
```
It seems the metrics are not saved together with the Checkpoint data and Early Stop tries to be applied straight away.
### To Reproduce
1. Start training a model with Checkpointing (best and last) and Early Stop on a validation metric. I have used Pytorch-Lightning Cli for this.
2. The model crashes after some checkpoints were saved
3. Try to re-run the training passing `ckpt_path` with the last checkpoint path to the trainer (with the same Early Stopping settings).
4. The error happens.
### Expected behavior
The model should resume training, knowing the last best value, and apply Early Stopping as appropriate.
### Environment
* CUDA:
- GPU:
- NVIDIA A100-SXM4-80GB
- available: True
- version: 11.4
* Packages:
- numpy: 1.21.2
- pyTorch_debug: False
- pyTorch_version: 1.10.0a0+0aef44c
- pytorch-lightning: 1.6.1
- tqdm: 4.62.3
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.8.12
- version: #103-Ubuntu SMP Fri Nov 26 16:13:00 UTC 2021
### Additional context
Running using the Pytorch Lightning Cli
cc @carmocca @awaelchli @rohitgr7
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with checkpoint restoration, Early Stopping, and a validation metric such as `val/f1/default`, then inspect how checkpoint state and EarlyStopping state are restored. Confirm the fix by resuming from the last checkpoint after a crash and verifying that training continues and Early Stopping uses the restored best value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100