Lightning-AI / Lightning-AI/pytorch-lightning
`lr_finder` fails when called after training for 1 or more epochs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🐛 Bug
Calling [`lr_finder`](https://pytorch-lightning.readthedocs.io/en/latest/api/pytorch_lightning.tuner.lr_finder.html?highlight=lr_finder) on the model after `trainer.fit()` has been called will fail with:
```
LR finder stopped early due to diverging loss.
Failed to compute suggesting for `lr`. There might not be enough points.
```
, even when the default value of `min_lr=1e-08` has been changed to `1e-30`.
## Please reproduce using [the BoringModel and post here](https://colab.research.google.com/drive/1HvWVVTK8j2Nj52qU4Q4YCyzOm0_aLQF3?usp=sharing)
- Reproduced using a **callback**: https://colab.research.google.com/drive/1sbOPs8edyFi_idJNnd6gr3etyv7V57YU?usp=sharing
- Reproduced with **calling Trainer twice**: https://colab.research.google.com/drive/1WxUvayBBg_163nu8fjv-jsvPk6pUrSrK?usp=sharing
### To Reproduce
Add the following callback (as demonstrated with the BoringModel):
```
# Call Learning Rate finder after X epochs
class LRFinderXEpoch(Callback):
def __init__(self, epoch=1):
super().__init__()
self.epoch = epoch
def on_train_epoch_start(self, trainer, pl_module):
if trainer.current_epoch == self.epoch:
print("Calling learning rate finder!")
trainer.tune(pl_module)
# trainer.tuner.lr_find(pl_module, min_lr=1e-30)
```
### Expected behavior
Find the best learning rate after a few epochs of training (e.g. when doing Transfer Learning).
### Environment
```
* CUDA:
- GPU:
- Tesla T4
- available: True
- version: 10.1
* Packages:
- numpy: 1.18.5
- pyTorch_debug: True
- pyTorch_version: 1.7.0+cu101
- pytorch-lightning: 1.0.8
- tqdm: 4.41.1
* System:
- OS: Linux
- architecture:
- 64bit
-
- processor: x86_64
- python: 3.6.9
- version: #1 SMP Thu Jul 23 08:00:38 PDT 2020
```
### Additional context
Issue came from the following discussion: https://forums.pytorchlightning.ai/t/train-2-epochs-head-unfreeze-learning-rate-finder-continue-training-fit-one-cycle/366/4
Potentially related issues:
- https://github.com/PyTorchLightning/pytorch-lightning/issues/4784
- https://github.com/PyTorchLightning/pytorch-lightning/issues/4616
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the BoringModel reproducer and the LRFinderXEpoch callback, then compare trainer.tune(pl_module) with trainer.tuner.lr_find after Trainer.fit has run for one or more epochs. The change is complete when the learning-rate finder succeeds after resumed training, including with the documented callback scenario, and a regression test covers it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100