Lightning-AI / Lightning-AI/LitServe
Prometheus & OpenTelemetry Metrics for LitServe
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.9k
- Forks
- 304
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 6
Description
## Problem
LitServe currently provides a foundation for logging via `loggers.py`, but production-ready Prometheus metrics are left to the user. Since LitServe is a multi-process system (multiple API servers + multiple inference workers), implementing metrics correctly is non-trivial.
### Current State
LitServe handles logging through the `Logger` base class and `_LoggerConnector`.
**Architecture**
* A dedicated logger process is started when `loggers=[MyLogger()]` is passed to `LitServer`.
* A `multiprocessing.Queue` is created.
* Inside `LitAPI.predict()`, calling `self.log("metric", value)` pushes `(metric, value)` into the queue.
* The logger process consumes the queue and calls `logger.process(metric, value)`.
* The `Logger.mount(path, app)` hook allows mounting an ASGI app (e.g. a `/metrics` endpoint) onto the main FastAPI application.
### What's Missing
* **No auto-instrumentation:** HTTP request count, request latency, error rate, queue depth, etc. must all be implemented manually.
* **Multiprocessing issue:** If `mount("/metrics", prometheus_app)` is used, the endpoint is served by the API server processes, while metrics updated in `logger.process()` live only in the logger process. As a result, the API servers cannot expose those metrics.
* **No standard integration:** Every user has to write their own `prometheus_client` integration and handle multiprocessing concerns themselves.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading loggers.py, the Logger base class, _LoggerConnector, Logger.mount, and the LitAPI.predict() logging path. Trace how the multiprocessing queue and logger process interact with the API server processes, then define the required Prometheus and OpenTelemetry integration and multiprocessing behavior. Done should include a concrete standard integration that exposes correct metrics from the server architecture.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fastapi, prometheus, python
- Domain
- api, backend-api-design, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100