apache / apache/tooling-llmao

Move fleet state into LiteLLM, enable STORE_MODEL_IN_DB

Open
#11 3 comments 0 reactions 1 assignee Claimed by @gstein View on GitHub
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Fleet membership lives in `config.yaml` → `fleet.hosts`. LiteLLM already has a
place for all of it, and moving it there means no second datastore.

A route carries everything: `litellm_params.api_base` is host and port,
`litellm_params.api_key` is the vLLM bearer token, and `model_info` is an
arbitrary dict for the recipe and provenance.

`GET /vllm/config` becomes: select routes whose `api_base` host matches the
caller's IP, return their `model_info.vllm` blocks and ports.

**`STORE_MODEL_IN_DB` is an environment variable**, not a config key. Without
it `/model/new` returns 500 with `Set 'STORE_MODEL_IN_DB='True'' in your env`.
Verified against the running proxy.

```
STORE_MODEL_IN_DB=True
```

The YAML `model_list` then becomes a bootstrap seed.

**An earlier proposal for a runtime-owned `fleet.yaml` is withdrawn.** It was
justified by a GPU box being able to fetch its assignment during a database
outage — but in that outage LiteLLM cannot authenticate or route either, so the
box comes up serving a model nothing can reach. The resilience buys nothing.

**Verified against litellm 1.99.0 with a local proxy.** `model_info` survives
a `/model/new` round-trip intact — nested dicts, arrays and integers all
preserved, with `id` and `db_model` added alongside rather than replacing
anything:

```json
"model_info": {
"vllm": {"args": ["--reasoning-parser","gemma4"],
"model": "google/gemma-4-26B-A4B-it",
"max_model_len": 131072},
"label": "roundtrip", "provider": "vast",
"id": "da959582-...", "db_model": false
}
```

**But `litellm_params` comes back encrypted.** `model`, `api_key` and
`api_base` are all ciphertext on read:

```json
"litellm_params": {
"model": "HrVkaVD0xIeeJYp9CNTwYWF7Yvm0jFF6t4Fa0oWa7xja...",
"api_key": "6dPA0ODCDmCuHal7fKy8tQjiobXMOgHdrNqhZjSI...",
"api_base": "Wdja0qDzWSaLfadF1Wdv85gNtx7E0NT05Ukc8fwv..."
}
```

That is LiteLLM protecting credentials at rest, which is right — but it means
**`api_base` cannot be read back to determine which host a route belongs to.**

`GET /vllm/config` matches routes by the caller's IP, so it needs that value.
Two options:

- decrypt via LiteLLM's own accessor, if one is exposed
- **duplicate the host into `model_info`**, where it stays plaintext
The second is simpler and keeps `/vllm/config` a plain query. It does mean the
host appears twice, so a route rewrite must update both — worth a helper rather
than two call sites.

Decide this before building, since it is a small schema choice with a large
debugging cost if missed.

See `docs/fleet-state.md` §1.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.