Work plan - Sep 02

# llmao — work plan

One ordered list. Each entry is a sub-issue of this issue; a `review` sub-issue explains
the reasoning behind them and does not sequence anything.

Phases are ordered by dependency, not by size. **p6 is a parallel track** and
can start at any point.

---

## Phase 0 — make it work at all (DONE)

Nothing routes through the gateway today. All four are small.

| | why now |
|---|---|
| **Stopgap: add api_base to model_list.yaml** | two lines; models answer today rather than 401ing to OpenAI |
| **AuthzError re-renders the page template** | authz failures return a full traceback including the session dict. Also unblocks `/fleet`, which is where skew is visible |
| **max_model_len absent from the example catalog** | one line per entry; removes a shipped hang condition |
| **LiteLLM proxy publicly reachable over plaintext** | vhost change: proxy `/v1/*` to loopback, close 4000 |

The first is deliberately a stopgap. It is replaced in phase 2, not extended.

---

## Phase 1 — trust the health signal before building on it

Phase 2 hangs entirely off health transitions: a route is created on the
healthy edge and deleted on the down edge. If those fire wrongly, it either
withholds a working server or pulls a live one from rotation.

| | why now |
|---|---|
| **Fleet health code is untested against real hardware** | described as "a rough v1 from Grok… needs to be tested in reality". Exercise a box that never comes up, one that dies, and one that flaps |
| **Surface config skew on the /fleet page** | `srv.skew` is already populated and logged; nothing displays it. This is why the routing mismatch went unnoticed. Needs the authz fix from phase 0 |
| **`db_model: false` on a route created via /model/new** | cheap to pin down, and phase 2 needs to distinguish llmao-created routes from seeded ones |
| **Fail fast on an unmapped port, and report the mapping back** | an unmapped port produces a healthy server nobody can reach. Phase 2 would register a route for it and fail on every request |

Skipping this phase is possible. It means finding out in phase 2 whether the
transitions were right, with more moving parts in the way.

---

## Phase 2 — fleet state into LiteLLM

**One commit, three parts.** Landing any part alone puts routes with no
`api_base` back into the proxy — which is the phase 0 outage, reproduced.

| | |
|---|---|
| **Move fleet state into LiteLLM, enable STORE_MODEL_IN_DB** | routes carry host, port, key and recipe |
| **Health-gated route registration** | a route exists only while its vLLM is serving |
| **Remove the model_list include from litellm.yaml** | otherwise LiteLLM reads catalog entries as routes |

Verified already: `model_info` survives a `/model/new` round-trip intact.
`litellm_params` reads back **encrypted**, so `api_base` cannot identify a
route's host — duplicate the host into `model_info` or decrypt via LiteLLM's
accessor. Decide that before writing code.

Also settle where pending assignments live between "host added" and "vLLM
healthy". Preferred: a small table in the same Postgres.

`model_list.yaml.example` in this packet is the post-change version. **Do not
copy it before this phase** — it has no `api_base`, which is correct afterwards
and broken before.

---

## Phase 3 — sizing and observability

Independent of each other; any order.

| | why |
|---|---|
| **Fit validation from model requirements** | `validate_fleet` does not check a placement fits. A percentage cannot be right on two card sizes |
| **Report observed vLLM state back to the control plane** | the box knows its measured KV cache and actual GPU; llmao never hears it. Every sizing problem so far was diagnosed by reading a startup log by hand |
| **Use --kv-cache-memory instead of a utilisation percentage** | vLLM prints the exact byte count. Measured beats derived |

---

## Phase 4 — UI

| | |
|---|---|
| **Fleet management in the UI** | add, retire, edit. Needs phase 2 |

---

## Parallel track — p6

Can start now. Nothing above blocks it, and it blocks nothing above.

| | |
|---|---|
| **Puppet manifest for the llmao gateway host** | portal as a `pipservice`, LiteLLM as a small manifest, Postgres from `base_packages` |
| **Backup for the LiteLLM database** | `$dest` is unresolved — confirm what BackupPC actually captures |

Two open questions in `p6/NOTES.md`: whether `update_pipenv` handles `uv` (the
script suggests yes), and the `/opt/llmao/llmao.yaml` vs `config.yaml`
filename, being resolved by renaming.

---

## Later — not yet sequenced

Both need a decision before they need code.

| | |
|---|---|
| **Budgets exist but meter nothing for self-hosted models** | the endpoints work; `spend_usd` is always 0.00 because self-hosted models have no cost map. Needs a decision on what a self-hosted token costs before it needs an implementation |
| **MCP server registration through llmao** | LiteLLM's MCP management already carries access groups, per-tool cost, and an admin-review workflow that matches llmao's existing PMC gate. Confirm whether MCP access is scoped per key or per project before building |

---

## Whenever

Small, unblocked, no dependencies.

| | |
|---|---|
| **`make db` rotates the Postgres password on every run** | running it twice silently breaks a working `litellm.yaml` |
| **Rotate the self-hosted vLLM API key** | pasted around during debugging; never committed |
| **SSL_VERIFY=0 is a stopgap that will outlive its reason** | file it or make it warn |
| **Two instances of one model do not need artificial model names** | documentation; prevents a worse catalog shape |

---

## The one thing to get right

Phase 2 is a single change. The temptation is to land the route push first and
tidy up afterwards — that reproduces the outage on every restart, and it will
look like a regression rather than an incomplete migration.