Lightning-AI / Lightning-AI/pytorch-lightning
Proper support for Pytorch SequentialLR Scheduler
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🐛 Bug
Currently there is a bug when a ReduceLROnPlateau is used inside SequentialLR due to no proper support for this scheduler in [Trainer._configure_schedulers](https://github.com/PyTorchLightning/pytorch-lightning/blob/b57feccbff57271144dae70a82b1948f7a4cf7af/pytorch_lightning/trainer/optimizers.py#L123-L193). An exception is raised since the monitor metric is not properly passed to the ReduceLROnPlateau scheduler in [TrainingEpochLoop._update_learning_rates](https://github.com/PyTorchLightning/pytorch-lightning/blob/b57feccbff57271144dae70a82b1948f7a4cf7af/pytorch_lightning/loops/epoch/training_epoch_loop.py#L447-L507)
Note: Currently, there is a bug in SequentialLR missing an optimizer attribute, see https://github.com/pytorch/pytorch/pull/67406 and https://github.com/PyTorchLightning/pytorch-lightning/issues/10278. But that should not interfere here.
### To Reproduce
run any lightning model with trainer with scheduler setup like:
```python
def configure_optimizers(self):
optimizer = torch.optim.Adam(self.parameters(), lr=0.01)
s1= torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer)
s2= torch.optim.lr_scheduler.ConstantLR(optimizer)
scheduler = torch.optim.lr_scheduler.SequentialLR(
optimizer,
schedulers=[s1, s2],
milestones=[2]
)
scheduler.optimizer = optimizer # dirty fix for bug in SequentialLR
return {"optimizer": optimizer,
"lr_scheduler": scheduler,
"monitor": "loss"}
```
### Expected behavior
The monitor value should be passed to the underlying ReduceLROnPlateau scheduler.
This is defenitely tricky to achieve as the current way is assuming a fixed scheduler setup for the entire training time, e.g. allows for multiple scheduler but if scheduler1 is changing midway it only works if it is not ReduceLROnPlateau.
### Environment
- PyTorch Lightning Version (e.g., 1.5.0): 1.5.3
- PyTorch Version (e.g., 1.10): 1.10
- Python version (e.g., 3.9): 3.9
- OS (e.g., Linux): Windows
- CUDA/cuDNN version: 10.2
- GPU models and configuration: -
- How you installed PyTorch (`conda`, `pip`, source): pip
- If compiling from source, the output of `torch.__config__.show()`: -
- Any other relevant information: -
### Additional context
cc @tchaton
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in Trainer._configure_schedulers in pytorch_lightning/trainer/optimizers.py and TrainingEpochLoop._update_learning_rates in pytorch_lightning/loops/epoch/training_epoch_loop.py. Reproduce with the SequentialLR configuration using ReduceLROnPlateau and the reported monitor value of "loss". Done means the underlying ReduceLROnPlateau scheduler receives the monitor value without raising an exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100