Lightning-AI / Lightning-AI/pytorch-lightning
LearningRateMonitor doesn't log to MLFlow via mlflow.pytorch.autolog()
@justusschock is already working on this.
Since Jun 10, 2021.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🐛 Bug
P-L doesn't send lr data to MLFlow when mlflow.pytorch.autolog is enabled and LearningRateMonitor callback activated
### To Reproduce
```
class EmbLt(pl.LightningModule):
def __init__(self, ...)
def configure_optimizers(self):
optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, self.parameters()), lr=self.learning_rate)
lr_scheduler = {
'scheduler': torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, EPS // 3, 1),
'name': 'lr_sched'
}
return [optimizer], [lr_scheduler]
lr_monitor = LearningRateMonitor(logging_interval='epoch')
emb = EmbLt(ntoken=len(vect_obj), input_size=emsize, hidden_size=nhid, learning_rate=lr, drop=drop_rate)
trainer = pl.Trainer(gpus=1, max_epochs=EPS, progress_bar_refresh_rate=20, callbacks=[lr_monitor])
with mlflow.start_run() as run:
trainer.fit(emb, dataloaders['tr'], dataloaders['ts'])
```
### Expected behavior
Dedicated log object inside MLFlow' current run
### Environment
* CUDA:
- GPU:
- NVIDIA Quadro RTX 5000
- available: True
- version: 11.0
* Packages:
- numpy: 1.20.3
- pyTorch_debug: True
- pyTorch_version: 1.7.0+cu110
- pytorch-lightning: 1.3.4
- tqdm: 4.51.0
* System:
- OS: Linux
- architecture:
- 64bit
-
- processor: x86_64
- python: 3.7.10
- version: #1 SMP Sun Feb 14 18:10:38 EST 2021
### Additional context
Was able to override it via this trick:
```
class EmbLt(....)
....
def on_epoch_start(self):
eopt = self.optimizers()
self.log('lr_curr', eopt.param_groups[0]['lr'])
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.