Lightning-AI / Lightning-AI/LitServe

health probes such as startupz, readyz

Open
#628 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
3.9k
Forks
304
Avg merge
3d 13h
Merged PRs (30d)
6

Description

## 🚀 Feature

liveness probe endpoints: startupz and readyz endpoint, specifically readyz when workers are ready

### Motivation

When deploying LitServe in production (e.g., within Kubernetes), we need built-in liveness and readiness probe endpoints to manage pod lifecycle properly.

Currently, the main process and worker processes have no standardized way to report their health states. Since model loading (setup()) happens in separate worker processes, the main process cannot easily expose a meaningful readiness signal. This causes Kubernetes to mark pods as ready before workers are actually able to serve inference requests.

### Pitch

Add built-in endpoints such as:
• /startupz — returns 200 when the LitServer process has successfully started.
• /readyz — returns 200 only when all worker processes have completed their setup() routines and are ready to serve requests.

These endpoints would allow production orchestrators (e.g., Kubernetes) to safely manage startup, readiness, and liveness of LitServe pods without requiring custom inter-process signaling.

### Alternatives

### Additional context

Contributor guide

Open the contributing guide

Research direction

Start with the LitServe server startup flow, worker processes, and their setup() routines, then trace where HTTP endpoints are registered. Define how startup and readiness state is communicated across workers; done means /startupz and /readyz return the requested statuses, with /readyz waiting until all workers can serve inference requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
api, backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.