Health-gated route registration
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
A provisioned instance can take fifteen minutes to load weights. A route whose
backend is not serving fails every request routed to it.
**Rule: a route exists in LiteLLM if and only if its vLLM is serving.**
```
add host -> record assignment, generate api_key, no route yet
box boots -> GET /vllm/config
health OK -> POST /model/new
health DOWN -> POST /model/delete
retire -> delete route, drop assignment
```
Uniform, no special cases. Generating the `api_key` at add-time and storing it
in `litellm_params.api_key` also answers where per-instance keys live.
**Why cooldown cannot substitute.** Verified against litellm 1.99.0:
- **No per-deployment enable/disable.** No `enabled` or `is_disabled` field,
nothing in the model-management endpoints. Registered or deleted, no third
state.
- **`DEFAULT_COOLDOWN_TIME_SECONDS` is 5**, so a cooled deployment re-enters
rotation almost immediately.
- **Single-deployment model groups are exempt.**
`SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD` is 1000, with the comment *"by
default we should avoid cooldowns on single deployment model groups."*
So the first server for a new model would fail every request for the whole boot
window, uncooled.
**Pending assignments need somewhere to live** — an IP, port, model name and
key per pending host. Preferred: a small table in the same Postgres. Not
LiteLLM's schema, but the same database, so no new backup story.
`cooldown_time` and `allowed_fails` are settable per-deployment via
`model_info` if different behaviour is wanted later.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the host add, boot health check, and retire flow around LiteLLM's /model/new and /model/delete endpoints. Design the pending-assignment table in the same Postgres database, including the per-instance API key, and verify that routes exist only while vLLM health is OK and are removed when it is not.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- api, backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100