Lightning-AI / Lightning-AI/pytorch-lightning

LearningRateMonitor doesn't log to MLFlow via mlflow.pytorch.autolog()

Open
#7,911 3 comments 0 reactions 1 assignee View on GitHub

@justusschock is already working on this.

Since Jun 10, 2021.

logger logging priority: 2 working as intended
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🐛 Bug

P-L doesn't send lr data to MLFlow when mlflow.pytorch.autolog is enabled and LearningRateMonitor callback activated

### To Reproduce

```
class EmbLt(pl.LightningModule):
def __init__(self, ...)

def configure_optimizers(self):
optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, self.parameters()), lr=self.learning_rate)
lr_scheduler = {
'scheduler': torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, EPS // 3, 1),
'name': 'lr_sched'
}
return [optimizer], [lr_scheduler]

lr_monitor = LearningRateMonitor(logging_interval='epoch')
emb = EmbLt(ntoken=len(vect_obj), input_size=emsize, hidden_size=nhid, learning_rate=lr, drop=drop_rate)
trainer = pl.Trainer(gpus=1, max_epochs=EPS, progress_bar_refresh_rate=20, callbacks=[lr_monitor])

with mlflow.start_run() as run:
trainer.fit(emb, dataloaders['tr'], dataloaders['ts'])
```

### Expected behavior

Dedicated log object inside MLFlow' current run

### Environment

* CUDA:
- GPU:
- NVIDIA Quadro RTX 5000
- available: True
- version: 11.0
* Packages:
- numpy: 1.20.3
- pyTorch_debug: True
- pyTorch_version: 1.7.0+cu110
- pytorch-lightning: 1.3.4
- tqdm: 4.51.0
* System:
- OS: Linux
- architecture:
- 64bit
-
- processor: x86_64
- python: 3.7.10
- version: #1 SMP Sun Feb 14 18:10:38 EST 2021

### Additional context

Was able to override it via this trick:
```
class EmbLt(....)
....

def on_epoch_start(self):
eopt = self.optimizers()
self.log('lr_curr', eopt.param_groups[0]['lr'])
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.