review

# tooling-llmao review

Read the whole thing. Short version: the architecture is right and the code is
better than the version it replaces. The sizing gap you flagged is real, and
there is a second gap next to it that I think matters more.

---

## What is good

**The control-plane split is the right call.** Boxes fetch `GET /vllm/config`
by client IP and write Supervisor units. Placement lives in one place, GPU
templates are identical, and adding a box is a config edit. That directly fixes
the thing that cost the most operational time on the current setup — every host
migration was a hand-run `vllm serve` plus a hiera edit plus a CDN ticket.

**Supervisor instead of `nohup`.** The current Gemma host has lost its server
silently more than once. `autorestart` is the fix, and using the platform's own
process manager rather than a Python one is correct.

**`validate_fleet` fails fast at startup** — unknown model, duplicate port,
duplicate name, missing `api_key`, missing `model_info.vllm`. Most of the
config-drift bugs in the current stack would have been caught by this.

**Health probing hits vLLM `/health`, not LiteLLM `/health`.** The design doc
says why, and it is right: LiteLLM's health check runs real completions and
costs tokens. Using it as a boot probe would be expensive and slow.

**The catalog carries `thinks_by_default`.** Someone measured that Gemma and
Qwen differ and wrote it down. That is exactly the kind of thing that is
invisible until it bites.

---

## The sizing gap

`model_info.vllm.gpu_memory_utilization` is a **hand-written constant** —
0.55 for Gemma, 0.40 for Qwen. `Server.from_row` reads it, `box_json` passes it
through, `install_set.py` writes it into the Supervisor unit. Nothing computes
it, and nothing checks it.

Three consequences, in increasing order of severity.

### 1. `validate_fleet` does not check that a placement fits

You can put four models summing to 2.1 utilisation on one box and the control
plane will happily serve that JSON. The box then fails at engine init with a
message that does not mention memory. This is the cheapest thing to fix and
catches the whole class.

### 2. A utilisation percentage cannot be correct on two different cards

0.55 of an 80 GB A100 is 44 GB — right for Gemma at BF16. 0.55 of a 48 GB L40S
is 26 GB — not enough, and it will fail. The same catalog entry cannot be right
on both, and `fleet.hosts` records only IP and port.

The fix is **not** to declare card size per host — see build item 3. It is to
put the model's absolute requirement in the catalog and let the box decide fit
against hardware it can actually see.

### 3. KV cache is measured, not configured

This is the one that produces silent failures rather than loud ones. vLLM
reports at startup:

```
GPU KV cache size: 855,962 tokens
Maximum concurrency for 131,072 tokens per request: 6.53x
```

**A `--max-model-len` above the measured cache produces hangs, not errors.** No
amount of static arithmetic tells you the real number — quantisation, attention
backend, CUDA-graph reservation and vLLM version all move it. On the current
fleet: Gemma BF16 on an 80 GB card gets 856k tokens of cache; Qwen FP8 on a
48 GB card gets 217k. Neither is predictable from the catalog alone.

---

## What I would build, in order

### 1. Move fleet state into LiteLLM — one to two days

Fleet membership lives in `config.yaml` → `fleet.hosts`. LiteLLM already has
somewhere for all of it, and using that means no second datastore.

A route carries `litellm_params.api_base` (host and port),
`litellm_params.api_key` (the vLLM bearer token), and `model_info` (an
arbitrary dict for the recipe and provenance). `GET /vllm/config` becomes a
query over routes matching the caller's IP.

Needs `STORE_MODEL_IN_DB=True` — an **environment variable**, not a config key.
Verified: without it `/model/new` returns 500.

**An earlier draft of this review proposed a runtime-owned `fleet.yaml`.** It
was justified by a GPU box being able to fetch its assignment during a database
outage. That does not hold: in such an outage LiteLLM can neither authenticate
nor route, so the box comes up serving a model nothing can reach.

**Verify first:** whether `model_info` survives a `/model/new` round-trip
intact.

### 2. `model_list.yaml` is the catalog, not the routing table

Once routes come from the database, `litellm.yaml`'s `include: model_list.yaml`
is actively harmful — LiteLLM reads catalog entries as routes, creates them
without an `api_base`, and sends every call to `api.openai.com`. That is the
outage in #1, reproduced on every restart.

The file becomes the **catalog**: what each model is, its licence and
provenance, and the vLLM recipe. `models.py:43` already treats it that way —
`public_models()` strips `litellm_params` and surfaces `model_info` as the UX.

`api_base` and `api_key` come out, since both are per-instance and generated at
registration. Nothing secret then remains in the file.

Must land with the two items around it, not after.

### Aside: N instances of one model need one `model_name`

Raised in discussion as *"we need to create artificial model names, so that we
can run several instances of (say) Qwen3."* That is not required.

**LiteLLM load-balances across deployments sharing a `model_name`.** That is
what a model group is — `router_utils/cooldown_handlers.py` treats
`len(model_group) == 1` as the special case precisely because more than one is
normal.

Five Qwen3 boxes with identical config are five routes, all `qwen3-8b`,
different `api_base`. Distinct names are needed only when the difference is
caller-visible — context window, reasoning parser, response shape.

The `[model, port, name]` third element is unrelated: it distinguishes two vLLM
processes on **one box**, which need separate ports and supervisor units.

The distinction matters because the two readings produce very different
catalogs — one where `qwen3-8b` is a single offering with N backends, and one
where callers pick between `-a`, `-b` and `-c` for no reason they can perceive.

### 3. Health-gated registration — one to two days

**A route exists if and only if its vLLM is serving.** A provisioned instance
takes fifteen minutes to load weights; a route registered earlier fails every
request routed to it.

Cooldown cannot substitute. Verified against litellm 1.99.0: there is no
per-deployment enable/disable, `DEFAULT_COOLDOWN_TIME_SECONDS` is 5, and
single-deployment model groups are explicitly exempt
(`SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD` = 1000). The first server for a
new model would fail every request for the whole window, uncooled.

Pending assignments need somewhere to live before the route exists — a small
table in the same Postgres is the least-bad option.

### 4. Fit validation: model requirements, not host declarations — half a day

**Host capacity should not be declared in config.** An earlier draft of this
review proposed `vram_gb: 80` per host; that is wrong. The box knows its own
VRAM from `nvidia-smi`, and a hand-typed number rots the first time a provider
hands you a different card than you paid for — which, with rented instances, is
a matter of when.

Put the **requirement** in the catalog, where it is a property of the
checkpoint and true everywhere:

```yaml
model_info:
vllm:
model: google/gemma-4-26B-A4B-it
vram_gb: 49 # BF16 weights, measured at load
disk_gb: 50
```

Then the box decides fit at install time, which is gstein's plan and the right
one: read actual free VRAM, subtract the requirement, compute what is left for
KV — or reject and report why.

Optionally carry `expect_gpu` / `expect_vram_gb` in `model_info` as a **hint
only**, so the UI can warn at add-time that Gemma will not fit on a 24 GB box
rather than letting you find out after it boots. Declared-versus-observed is
then a useful signal in its own right: you paid for an 80 GB card and got
something else.

### 5. Report observed state back to the control plane — one to two days

The box knows the truth and llmao never hears it. Have the install step POST
what it actually found:

```
POST /vllm/observed
{ "name": "gemma4-26b",
"gpu": "NVIDIA A100-SXM4-80GB", "vram_total_gb": 80.0,
"weights_gb": 48.5, "kv_cache_tokens": 855962,
"max_model_len": 131072, "attention_backend": "FLASH_ATTN",
"vllm_version": "0.25.1" }
```

Then `/fleet` can show, per server:

- **measured KV cache vs configured `max_model_len`** — red when max_model_len
exceeds the cache, which is the hang condition
- declared vs observed GPU, so a mis-provisioned rental is visible
- whether the fused-MoE config was found (vLLM warns when it is not; on the
current A100 it is absent for Gemma and performance is sub-optimal)

This is the item that would have saved the most time on the existing setup —
every sizing problem so far was diagnosed by reading a vLLM startup log by
hand.

### 6. Use `--kv-cache-memory` instead of deriving a utilisation percentage

An earlier draft proposed computing `gpu_memory_utilization` from weights and
headroom. **Do not build that.** vLLM already reports the exact figure:

```
Free memory on device (23.3/23.56 GiB) on startup. Desired GPU memory
utilization is (0.8, 18.85 GiB). Actual usage is 9.17 GiB for consumed memory
(weights + non-torch), 0.18 GiB for peak activation, and 0.62 GiB for
CUDAGraph memory. Replace gpu_memory_utilization config with
`--kv-cache-memory=9373714330` (8.73 GiB) to fit into requested memory, or
`--kv-cache-memory=14154919424` (13.18 GiB) to fully utilize gpu memory.
```

So the install step can start with a conservative utilisation, read the
suggested byte count out of the log, and relaunch with `--kv-cache-memory` set
absolutely. That is measured rather than derived, and it removes the whole
class of "the percentage was right for the other card".

It also fixes the underlying complaint that a percentage is meaningless without
knowing the card — on a 24 GB box you want 0.8, on an 80 GB box 0.3, and the
same catalog entry cannot be correct on both.

### 7. Fleet management in the UI — two to three days

With items 1–2 done, `/fleet` becomes the management surface. Three actions:

- **Add host** — IP, label, provider, instance id, `[model, port]` pairs.
Validate the model exists and the port is unclaimed, stamp `added` /
`added_by` from the session, write atomically, reload.
- **Retire host** — move to `retired` with a required reason. The box starts
getting 404 from `/vllm/config`, which is the correct signal that it is no
longer ours.
- **Edit servers** — change the model/port list for a box being repurposed.

Needs `Fleet.reload()` that **preserves health state for unchanged hosts**.
With a dozen hosts and a 1800s health grace, resetting every probe because you
added one box means twenty minutes of `starting` on nineteen healthy servers.

Depends on item 1 landing first — the UI cannot write a file Puppet overwrites.

## Separate from sizing, and I think more urgent

### `max_model_len` is not in the example catalog at all

`Server.from_row` reads `vllm.get("max_model_len")` and `box_json` passes it
through, but neither example model sets it. So vLLM defaults to the model's full
architectural window.

For Qwen3-8B that is fine — 40,960 is both the ceiling and the config value. For
Gemma it means 131,072 on a card whose KV cache may not cover it, on a host
whose capacity nobody declared. **That is the hang condition, shipped as the
default.**

At minimum, set it explicitly in the example catalog so it is a visible decision
rather than an implicit one.

### Fleet key is a single shared secret, and the design doc knows it

Section 3.1 argues operational simplicity and says it can be hardened later.
Fair. But note it now grants: read access to every model API key in the fleet,
from any IP that appears in `fleet.hosts`. One compromised box exposes the whole
set.

The mitigation that costs nothing is already half-there — the endpoint is keyed
by client IP, so a leaked key from an IP not in `fleet.hosts` gets nothing.
Worth stating that explicitly in the doc as the actual boundary, because it is
stronger than the doc gives itself credit for.

### `SSL_VERIFY=0` is a stopgap that will outlive its reason

Design doc section 5 notes it is temporary while `llm.apache.org` is on :8443.
Stopgaps in provisioning scripts do not get removed. Either file it as an issue
now, or have `install_set.py` log a warning when it is set, so it announces
itself.

---

## Smaller things

**`apptoken.txt` and `llmao-state.json` are in the repo.** Check whether they
are gitignored — asfquart writes the former, and a committed session key is
worth catching before this becomes the official gateway.

**`certs/*.pem` are committed.** Fine for localhost dev certs, but worth a
comment in `certs/README.md` saying so explicitly, because "why is there a
private key in the repo" is the kind of question that stalls a review.

**Health grace is 1800s.** That is right for a cold Gemma pull, and it is nice
to see it acknowledged rather than defaulted to something optimistic. Worth a
comment saying why, since 30 minutes looks absurd without the context.

**Test coverage is on the right things.** `test_fleet_health.py` and
`test_hosting.py` cover the box JSON and the Supervisor unit generation — the
two places where a mistake is expensive and invisible.

Worth noting the health implementation itself is described as *"a rough v1 from
Grok… needs to be tested in reality."* Everything in the fleet work hangs off
those transitions being correct, so exercising them against real hardware is a
prerequisite rather than a follow-up.

---

## Found while testing (2026-08-29)

Three defects in the committed `model_list.yaml`, found pointing a project key
at the deployed proxy. All three are in the file that is meant to be the single
source of truth for routes.

### T1 — `api_base` has a doubled `/v1` (see also T5)

```yaml
# model_list.yaml, gemma4-26b
api_base: https://llm.tooling.apache.org/v1 # wrong
api_base: https://llm.tooling.apache.org # right
```

LiteLLM appends `/v1/chat/completions` to `api_base`, so this resolves to
`/v1/v1/chat/completions`. The CDN 404s, LiteLLM cannot parse the response, and
its error handling reports an **OpenAI authentication failure** — the message
names `platform.openai.com` despite OpenAI never being contacted.

That misleading error is the worst part: the symptom points at the caller's key,
which is the one thing that is fine. The `qwen3-8b` entry two blocks down has
the correct shape (`http://127.0.0.1:8003`, no suffix) — worth a comment in the
example file, since one entry teaches the wrong pattern.

**Worth considering:** normalise in `Server.from_row` or at config load —
strip a trailing `/v1` and log when doing so. This is a one-character mistake
with a diagnosis that leads away from the cause.

### T2 — a live vLLM API key is committed in cleartext

`model_list.yaml` carries `api_key: 55e7c7fb081ef433...` for Gemma. The file
header says "Puppet/eyaml substitutes API keys only", which implies the
committed copy should hold placeholders — and the `qwen3-8b` entry still has
`CHANGE_ME_SELFHOST_API_KEY`, so the intent is clear and Gemma's is a mistake.

Three things:

1. **Rotate that key.** It is the vLLM bearer token for the Gemma endpoint. If
the file has been pushed, treat it as public.
2. **Confirm `model_list.yaml` is gitignored** (`.example` is the tracked one).
`git check-ignore -v model_list.yaml` — if it is not, that is the actual bug
and T2 will recur.
3. **Add a pre-commit or CI check** that fails on a non-placeholder `api_key`
in the tracked YAML. A regex for `api_key:` not matching `CHANGE_ME_` costs
nothing and closes the class.

Note `config.yaml` and `litellm.yaml` have the same shape (`.example` tracked,
live file substituted) and are worth the same audit. `apptoken.txt` and
`llmao-state.json` also appear in the repo listing.

### T3 — `max_model_len` is absent from both catalog entries

Already noted in the review body; repeated here because T1 surfaced it. Neither
`gemma4-26b` nor `qwen3-8b` sets it, so vLLM serves each model's full
architectural window regardless of what the host's measured KV cache supports.
For Gemma that is 131,072 on a card whose capacity is not declared anywhere.

This is the hang condition, shipped as the default, in the file that is the
source of truth.

### T4 — the LiteLLM proxy is publicly reachable over plaintext HTTP

`http://llm.apache.org:4000` serves `/v1/models` and `/v1/chat/completions`
without TLS. Project keys and prompt payloads — ASF source and draft security
findings — cross the internet in the clear.

This is the same exposure that the Fastly work closed for the Gemma endpoint
a few weeks ago, reintroduced one layer up.

Two fixes, either sufficient:

- **Proxy `/v1/*` through the portal vhost** to `127.0.0.1:4000`, and rebind
4000 to loopback. One TLS hostname, and it matches how gofannon and llmao
already reach each other.
- **Restrict 4000 to the tailnet.** Plain HTTP inside a private network is
defensible; the current gofannon deployment does exactly this with
`http://100.105.28.100:4000/v1`.

Worth confirming the public bind is intentional before assuming it needs
fixing — but if it is, it should be deliberate and documented rather than a
default.

---

### T5 — nothing reconciles the fleet with LiteLLM's routes

This is the structural one, and it subsumes T1. Found on
`tooling-llmao-ec2-or`, where the two halves describe different worlds.

**`config.yaml` fleet:**

```yaml
hosts:
192.220.55.116:
- [qwen3-8b, 8001]
166.88.36.242:
- [qwen3-8b, 8001]
```

**`model_list.yaml` routes:**

```yaml
- model_name: qwen3-8b
litellm_params:
model: openai/qwen3-8b
api_key: sk-573e6 # no api_base

- model_name: gemma4-26b
litellm_params:
model: openai/gemma4-26b
api_key: sk-9c466 # no api_base, and no fleet server at all
```

Three things wrong at once:

- **Neither route has `api_base`.** With `model: openai/…` and no base,
LiteLLM sends the request to **api.openai.com**, presents the `sk-` value,
and OpenAI rejects it. Every call to either model 401s with an error naming
`platform.openai.com` — which points the reader at their own key, the one
thing that is fine.
- **`gemma4-26b` has a route but no fleet server.** Nothing serves it.
- **`qwen3-8b` is placed on two hosts but has one route.** If that placement is
deliberate (two replicas for capacity) LiteLLM needs two entries to balance
across them. If it is a copy-paste while Gemma was meant to move to the
second host, that is a different bug.

**The code already knows how to compute the right answer.** `Server.api_base`
in `fleet.py:218` derives `http://:` from the placement. So the
fleet knows where every model runs — nothing writes that into LiteLLM's routes.

Worse, `check_config_skew` in `litellm_client.py:495` **detects exactly this**:

```python
bases = _api_bases_from_model_info(...) # empty -- no api_base anywhere
for srv in self.fleet.servers:
if srv.api_base in bases: ... # never true
elif note not in srv.skew:
_LOGGER.warning(f"skew: {srv.name}@{srv.api_base} missing from LiteLLM")
```

With `bases` empty, both fleet servers should log `missing from LiteLLM` every
`skew_interval_s` (180s). Nobody has seen those warnings. So the system
diagnosed itself correctly and the diagnosis went to a log nobody reads.

Note the `extra` branch below it — `bases - fleet_bases` — is vacuous in this
state. It only catches LiteLLM having routes the fleet does not know about,
which cannot trigger when no route has a base at all.

**The fix is generation, not editing.** Hand-adding `api_base` to both entries
works today and drifts the moment a model moves — which is the exact failure
the fleet design exists to prevent. Two shapes:

- **render `model_list.yaml` from the fleet** at startup, `api_base` derived
from `Server.api_base`, and treat hand edits to that field as unsupported
- **or push routes via LiteLLM's admin API** from the fleet, and stop keeping
`api_base` in the YAML at all

The second fits the existing design better, since `litellm_client` already has
an admin client and the fleet already owns placement.

**And skew should be visible, not just logged.** The `/fleet` page shows
server health; `srv.skew` is already populated by the check and is not
surfaced. A red row saying "missing from LiteLLM" would have made this
diagnosable in seconds rather than by tracing a misleading OpenAI error back
through three layers.

**Question for whoever owns the design:** is `model_list.yaml` intended to be
generated, hand-maintained, or hand-maintained-with-validation? All three are
defensible; the current state is the fourth option, where half is derived and
half is hand-written and the reconciliation is a warning nobody sees.

---

### T6 — AuthzError re-renders the page template, producing a 500 with a full traceback

`/fleet` returns a stack trace rather than a 403. The cause is not the fleet
page — it is how authorization failures are handled generally, and every route
that raises partway through a handler has the same shape.

```python
@APP.get("/fleet")
async def fleet_page(result):
if not result.is_site_admin:
raise AuthzError("site admin only") # line 217
...
result.litellm_ui = f"{litellm}/ui" # line 222 -- never reached
```

The `AuthzError` propagates, something catches it and flashes `site admin
only`, and then **the page template is rendered anyway**. `fleet.ezt` line 9
reads `[litellm_ui]`, which the aborted handler never assigned, and ezt raises
`UnknownReference`. Quart finds no handler for that and returns a 500 with the
full traceback — including the session dict, config paths, and the rendered
program.

Both halves are visible in the failure output: the flash is present *and* the
variable is missing.

```
'flashes': [{'category': 'warning', 'message': 'site admin only'}]
UnknownReference litellm_ui in fleet.ezt at line 9
```

**The code that sets `litellm_ui` is correct.** This is not a missing
assignment — it is an assignment that authz skipped.

`/keys/other` (line 341) and `/keys/other/new` have the identical pattern, so
they will fail the same way for a non-PMC user: a traceback where a 403 was
intended.

Two things wrong, in order of severity:

- **Disclosure.** An authorization failure returns more information than a
success would. The traceback carries the full session dict, filesystem paths,
and template internals to a user who was just told they lack access.
- **Diagnosis.** The error names `litellm_ui`, which sends the reader looking
for a missing variable rather than a failed permission check.

**Fix:** either render an error template that shares no variables with page
templates, or move authz into a decorator that runs before the handler so the
template is never reached. The second is cleaner — the check is already the
first statement in every affected handler, so it is decorator-shaped already.

**Note the two bugs stack.** `site_admins` is `[]` in `config.yaml.example`,
so a fresh deployment has no site admin at all and `/fleet` is unreachable by
anyone — which is why nobody noticed the page was broken. Worth documenting
that the first admin has to be added to `config.yaml` by hand; there is no
bootstrap path in the UI.

**And this is why T5 went unseen.** The `/fleet` page is the only surface
showing `srv.skew`, and it has been returning a 500 to everyone. The system
detected the route mismatch, wrote it to `srv.skew`, and the one page that
would have displayed it could not render.

---

## Sequencing

See `PLAN.md`. This document explains the reasoning; it deliberately does not
carry its own ordering, since two competing orders is worse than none.

---

# Appendix: making calls through llmao

The portal does not proxy chat. It mints **project-scoped virtual keys** and
you point tools at the LiteLLM proxy directly with one. Two endpoints, two
credentials, and conflating them is the most common source of 401s.

| | you present | to | for |
|---|---|---|---|
| portal | ASF OAuth session | `https://` | minting keys, budgets, usage |
| proxy | `sk-…` project key | `https://` | actual model calls |

## 1. Mint a key

In the portal: **Keys → Create personal key**, pick a project, give a purpose.
The `sk-…` value is shown **once** — copy it immediately, it is not
retrievable afterwards.

`/keys/new` is available to anyone signed in. `/keys/other/new` mints an
automation key not tied to a person, and is restricted to PMC members and site
admins — that is the one to use for a service like gofannon, since a personal
key dies with the person's access.

## 2. Check what you can reach

```bash
export LLMAO_KEY='sk-...'
export LITELLM_URL='https://'

curl -sS "$LITELLM_URL/v1/models" -H "Authorization: Bearer $LLMAO_KEY" \
| python3 -c 'import sys,json; print([m["id"] for m in json.load(sys.stdin)["data"]])'
```

Note this reports **what the key is permitted to use**, which is a snapshot
from when the key was minted. It is not the same as what the proxy can route
to. For that, an admin can query `/model/info` with the master key — worth
knowing when a model exists but a key cannot see it.

## 3. Make a call

```bash
curl -sS "$LITELLM_URL/v1/chat/completions" \
-H "Authorization: Bearer $LLMAO_KEY" \
-H 'Content-Type: application/json' \
-d '{"model":"gemma4-26b",
"messages":[{"role":"user","content":"Reply with exactly: OK"}],
"max_tokens":20,"temperature":0.6}' | python3 -m json.tool
```

Use the **bare** `model_name` from the catalog (`gemma4-26b`, `qwen3-8b`), not
a prefixed form. The `system_fingerprint` in the response identifies which vLLM
answered, which is how you confirm routing when several models are in play.

Thinking defaults differ per model and are recorded in the catalog:

```bash
# Qwen reasons by default; suppress it for a fast direct answer
-d '{"model":"qwen3-8b", ..., "chat_template_kwargs":{"enable_thinking":false}}'

# Gemma does NOT reason by default; ask for it explicitly
-d '{"model":"gemma4-26b", ..., "chat_template_kwargs":{"enable_thinking":true}}'
```

A thinking model at a low `max_tokens` returns `content: null` with the whole
budget spent in `reasoning_content`. That is not an error — raise `max_tokens`
or disable thinking.

## 4. Budget and usage

Against the **portal**, with an ASF session rather than the key:

```bash
curl -sS "https:///v1/projects/tooling/budget" -b "$SESSION_COOKIE"
# {"project":"tooling","provisioned":true,"max_budget_usd":100.0,
# "spend_usd":0.0,"remaining_usd":100.0}

curl -sS "https:///v1/projects/tooling/usage" -b "$SESSION_COOKIE"
# {"project":"tooling","entries":[...],"total_cost_usd":0.0,"count":0}
```

`/budget` requires committer role; `/usage` requires any signed-in identity.
`/healthz` is unauthenticated.

Self-hosted models have no cost map in LiteLLM, so `spend_usd` reads 0.00
regardless of tokens consumed. What you get from these endpoints is **request
attribution**, not dollars — worth stating before anyone builds billing on it.

## 5. Pointing gofannon at it

Two hiera values on `tooling-gofannon-ec2-va`, and **both must change
together**:

```yaml
# data/nodes/tooling-gofannon-ec2-va.apache.org.yaml
tooling::gofannon::llmao_api_base: 'https:///v1'
```

```yaml
# data/nodes/tooling-gofannon-ec2-va.apache.org.eyaml
tooling::gofannon::llmao_api_key: ENC[GPG,...] # the sk-… project key
```

Changing the base without the key gives a 401 that looks like a proxy fault.

```bash
puppet agent -t
docker exec gofannon-api env | grep LLMAO # base and sk-… key must match the file

docker exec gofannon-api sh -c 'curl -sS -m 30 "$LLMAO_API_BASE/chat/completions" \
-H "Authorization: Bearer $LLMAO_API_KEY" -H "Content-Type: application/json" \
-d "{\"model\":\"gemma4-26b\",\"messages\":[{\"role\":\"user\",\"content\":\"OK?\"}],\"max_tokens\":5}"'
```

Test the key from your laptop **before** wiring it in. A failure inside the
container could be the key, the base URL, or the network path; testing the key
independently removes one of the three.

gofannon sends `openai/gemma4-26b`; the LiteLLM SDK strips the provider prefix
client-side so the proxy receives the bare name. That is why the catalog's
`model_name` must be bare — a prefixed one 400s on an unknown model.

## Troubleshooting

| symptom | cause |
|---|---|
| `Malformed API Key … Ensure Key has 'Bearer ' prefix` | LiteLLM requires keys to start `sk-`; you sent a vLLM key or an empty value |
| `{"error":"Unauthorized"}` | vLLM's error format — you reached a model server directly, not the proxy |
| fast 403 on one model only | the key's allow-list predates a model rename; re-mint |
| 400 unknown model | prefixed name sent, or catalog `model_name` and proxy disagree |
| 503 from the proxy | the backing vLLM is down — check the fleet page |
| `content: null` | thinking consumed the whole `max_tokens` budget |
| `spend_usd` always 0.00 | expected for self-hosted models; no cost map |

This review spans several proposed changes rather than one bounded task. Start with models.py:43 and trace the /model/new, /vllm/config, and /fleet entry points; select and split out one item before implementation. Done should be defined by the chosen item’s stated verification, such as preserving model_info on a round-trip or exposing observed fleet state.

Written by the indexing model from the issue text.