Lightning-AI / Lightning-AI/pytorch-lightning

Log default metrics

Open
#20,418 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature logger
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

When training a model, I have to specify dataloaders, epochs, learning rate and I would like them to be logged by default (like huggingface).
(Could be a DeviceStatMonitor + Batch throughput + dataset metrics)

### Pitch

When training a model, many metrics are accessible and it would be really nice to log them directly like :

Pseudo-code
```python
def Trainer.fit():
for metric in ["learning_rate", "train_dataloader_len", "precision", "epochs", "limit_batches", ...]:
for logger in logger_that_has_been_set:
logger.log(metric, value)
```

### Alternatives

Log all the metrics myself for all the loggers like defined above (long and tedious...), a flag like "log_default_metrics=True" would be a nice alternative.

### Additional context

I'm using Databricks (mlflow) and I can have my custom model metrics but nothing in the system metrics or default parameters, default model metrics.

Thanks for the framework, it's really nice !

cc @borda @awaelchli

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing Trainer.fit and the configured logger path described in the issue, then review how DeviceStatMonitor, batch throughput, and dataset metrics could fit. Done should mean an agreed set of default training metrics is logged to every configured logger, but the issue names no files or tests and needs scope clarification.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.