Lightning-AI / Lightning-AI/pytorch-lightning

Support Lightning Logging without Trainer

Open
#8,509 10 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

design discussion feature logging priority: 2
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🚀 Feature

### Motivation

To ease conversion from pure PyTorch to Lightning, users might start by creating their LightningModule.

However, their code would break if they try to log as the trainer isn't available.

Currently, we have 2 options:
* make `self.log` in the absence of a Trainer
* add support for logging without a trainer.

Here is a pseudo code to explain how we could support it.

The `ResultCollection` object is pretty self contained and is used to store logged values.

```py
class LightningModule:
def __init__(self):
self._lightning_results = ResultCollection
self.training_step = self._training_step_wrapper(self.training_step)

@property
def _results(self):
if getattr(self, "trainer", None) is not None:
return self.trainer._results
return self._lightning_results

def _training_step_wrapper(self, training_step_fn):
def wrapper(self, *args, **kwargs)
self._current_fx = "training_step"
output = training_step_fn(self, *args, **kwargs)
self._current_fx = None
return wrapper

def training_step():
self.log(...)

class Model(LightningModule):
...

model = Model()

for _ in range(epochs):
for batch in datalaoder:
loss = model.training_step(batch, batch_idx)
...
logged_metrics = model.get_logged_metrics()

reduced_metrics = model.get_callback_metrics(epoch=True)
```

Drawback, every LightningModule hooks used for logging should be wrapped to set the `_current_fx` function.

### Pitch

### Alternatives

### Additional context

#### If you enjoy PL, check out our other projects:

- [Metrics](https://github.com/PyTorchLightning/metrics): Machine learning metrics for distributed, scalable PyTorch applications.
- [Flash](https://github.com/PyTorchLightning/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning
- [Bolts](https://github.com/PyTorchLightning/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch
- [Lightning Transformers](https://github.com/PyTorchLightning/lightning-transformers): Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

cc @borda @tchaton @justusschock @awaelchli @rohitgr7 @akihironitta @carmocca @edward-io @ananthsub @kamil-kaczmarek @Raalsky @Blaizzy

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading LightningModule, its self.log path, and the ResultCollection object described in the issue. Trace how training_step and other hooks currently obtain trainer state, then define how standalone logging and get_logged_metrics/get_callback_metrics should behave. Done means logging works without a Trainer while preserving trainer-backed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.