Lightning-AI / Lightning-AI/pytorch-lightning
Log default metrics
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
When training a model, I have to specify dataloaders, epochs, learning rate and I would like them to be logged by default (like huggingface).
(Could be a DeviceStatMonitor + Batch throughput + dataset metrics)
### Pitch
When training a model, many metrics are accessible and it would be really nice to log them directly like :
Pseudo-code
```python
def Trainer.fit():
for metric in ["learning_rate", "train_dataloader_len", "precision", "epochs", "limit_batches", ...]:
for logger in logger_that_has_been_set:
logger.log(metric, value)
```
### Alternatives
Log all the metrics myself for all the loggers like defined above (long and tedious...), a flag like "log_default_metrics=True" would be a nice alternative.
### Additional context
I'm using Databricks (mlflow) and I can have my custom model metrics but nothing in the system metrics or default parameters, default model metrics.
Thanks for the framework, it's really nice !
cc @borda @awaelchli
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing Trainer.fit and the configured logger path described in the issue, then review how DeviceStatMonitor, batch throughput, and dataset metrics could fit. Done should mean an agreed set of default training metrics is logged to every configured logger, but the issue names no files or tests and needs scope clarification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100