Lightning-AI / Lightning-AI/pytorch-lightning

Runtime events with support for custom handlers

Open
#8,895 9 comments 4 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

feature help wanted let's do it! logging
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🚀 Feature

### Motivation

For large scale Lightning deployments, it is very useful to collect certain types of trainer runtime event into an analytics system, so engineers can monitor the overall healthiness of the deployment, as well as obtain useful information for troubleshooting failed/stuck jobs.

Currently, Lightning mainly relies on logging for communicating trainer runtime information to the users. However, logging is not ideal for large scale deployments for the following reasons:
- Many training jobs in large scale deployments are automated, long-running batch jobs. It is impractical to expect the users to notice certain warnings (e.g. API deprecation warning) in a timely fashion.
- The desirable amount of events for debuggability purposes may be too verbose for logging.
- Logging is generally less structured.

### Pitch

- Introduce the concept of runtime event in Lightning
- Emit appropriate runtime events at appropriate places (e.g. deprecated API usage)
- Provide a mechanism to register custom handlers for runtime events
- Ideally, we'd like a mechanism to transparently register default handlers (unlike `Plugin` which requires users to explicitly pass to `Trainer`) which don’t belong in the core trainer directly. This could make Lightning more attractive for larger organizations that relies on shared tooling.
- The backend registration mechanism of `fsspec` is an ideal candidate solution: https://filesystem-spec.readthedocs.io/en/latest/developer.html#implementing-a-backend

### Alternatives

### Additional context
A similar prior feature request: https://github.com/PyTorchLightning/pytorch-lightning/issues/8186

______________________________________________________________________

#### If you enjoy Lightning, check out our other projects! ⚡

- [**Metrics**](https://github.com/PyTorchLightning/metrics): Machine learning metrics for distributed, scalable PyTorch applications.

- [**Flash**](https://github.com/PyTorchLightning/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

- [**Bolts**](https://github.com/PyTorchLightning/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

- [**Lightning Transformers**](https://github.com/PyTorchLightning/lightning-transformers): Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.