Move fleet state into LiteLLM, enable STORE_MODEL_IN_DB
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Fleet membership lives in `config.yaml` → `fleet.hosts`. LiteLLM already has a
place for all of it, and moving it there means no second datastore.
A route carries everything: `litellm_params.api_base` is host and port,
`litellm_params.api_key` is the vLLM bearer token, and `model_info` is an
arbitrary dict for the recipe and provenance.
`GET /vllm/config` becomes: select routes whose `api_base` host matches the
caller's IP, return their `model_info.vllm` blocks and ports.
**`STORE_MODEL_IN_DB` is an environment variable**, not a config key. Without
it `/model/new` returns 500 with `Set 'STORE_MODEL_IN_DB='True'' in your env`.
Verified against the running proxy.
```
STORE_MODEL_IN_DB=True
```
The YAML `model_list` then becomes a bootstrap seed.
**An earlier proposal for a runtime-owned `fleet.yaml` is withdrawn.** It was
justified by a GPU box being able to fetch its assignment during a database
outage — but in that outage LiteLLM cannot authenticate or route either, so the
box comes up serving a model nothing can reach. The resilience buys nothing.
**Verified against litellm 1.99.0 with a local proxy.** `model_info` survives
a `/model/new` round-trip intact — nested dicts, arrays and integers all
preserved, with `id` and `db_model` added alongside rather than replacing
anything:
```json
"model_info": {
"vllm": {"args": ["--reasoning-parser","gemma4"],
"model": "google/gemma-4-26B-A4B-it",
"max_model_len": 131072},
"label": "roundtrip", "provider": "vast",
"id": "da959582-...", "db_model": false
}
```
**But `litellm_params` comes back encrypted.** `model`, `api_key` and
`api_base` are all ciphertext on read:
```json
"litellm_params": {
"model": "HrVkaVD0xIeeJYp9CNTwYWF7Yvm0jFF6t4Fa0oWa7xja...",
"api_key": "6dPA0ODCDmCuHal7fKy8tQjiobXMOgHdrNqhZjSI...",
"api_base": "Wdja0qDzWSaLfadF1Wdv85gNtx7E0NT05Ukc8fwv..."
}
```
That is LiteLLM protecting credentials at rest, which is right — but it means
**`api_base` cannot be read back to determine which host a route belongs to.**
`GET /vllm/config` matches routes by the caller's IP, so it needs that value.
Two options:
- decrypt via LiteLLM's own accessor, if one is exposed
- **duplicate the host into `model_info`**, where it stays plaintext
The second is simpler and keeps `/vllm/config` a plain query. It does mean the
host appears twice, so a route rewrite must update both — worth a helper rather
than two call sites.
Decide this before building, since it is a small schema choice with a large
debugging cost if missed.
See `docs/fleet-state.md` §1.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.