[Epic]: Drive the two-card fleet to a measured performance envelope

### Executive Summary

**This fleet has never been measured under load.** Every latency figure produced on 2026-09-03 — and every figure in the reports before it — is at **concurrency 1**. There is no goodput number, no SLO, and no measurement at the configured `--max-num-seqs 4`. We have been tuning a system whose throughput nobody has established.

Meanwhile three levers sit untried, and the arithmetic says each is worth a large multiple:

| Lever | Today | Expected | Cost |
|---|---|---|---|
| **FP8 weights** — decode is bandwidth-bound at 280 GB/s; measured ~92 tok/s ⇒ **~2.83 GiB read per token**. Halving the bytes halves the time | ~92 tok/s | **~184 tok/s** | a values change + a quality gate. The card has hardware FP8 (sm_89) |
| **Cap context 32k** — one request can currently claim a quarter of the whole KV pool | max concurrency **3.80×** | **~30×** | a values change ([ADR-0011](https://github.com/ADORSYS-GIS/inference-ops/pull/29)) |
| **Reclaim card 1** — it holds a GPU and serves nothing | 1 of 2 cards useful | 2 of 2 | [ADR-0007](https://github.com/ADORSYS-GIS/inference-ops/pull/29) |

This EPIC drives the deployment from *"it works"* to **a measured, defended performance envelope** — every gain attributable, every regression caught.

### Strategic Intent

Turn performance from an anecdote into a number we can defend, then move that number deliberately. Three of this repo's last four performance-shaped decisions were argued from derived figures that later turned out wrong; the discipline this EPIC installs is **measure, change one thing, re-measure**.

The wider goal it serves: the fleet is meant to host [self-hosted agent roles](https://github.com/ADORSYS-GIS/inference-ops/pull/29). Agent traffic is concurrent and bursty — exactly the regime we have never measured and are currently configured against (`--max-num-seqs 4`).

### Problem Statement

1. **No throughput baseline exists.** We cannot say whether a change helped.
2. **`--max-model-len 262144` caps concurrency at 3.80×.** vLLM's own startup log says so. The model demonstrably cannot use that window (AA-LCR 25.6 vs the 9B's 63.0; locally, it misreads at 18k what it reads correctly at 963 tokens).
3. **Weights are bf16 because they fit, not because that is fast.** Decode is bandwidth-bound. On a card with hardware FP8, serving bf16 leaves roughly half the decode throughput on the table.
4. **Half the GPU fleet serves nothing.** `z-image-turbo` holds a card with zero routes pointing at it.
5. **Every change costs a ~3.5 min outage** (`Recreate`, one GPU, one replica), so unmeasured changes are expensive to undo — which is an argument for measuring, not for standing still.

### Desired Outcome

A committed, reproducible **goodput curve** for the fleet at a stated SLO, a decode rate improved by a measured multiple, and a regression gate that fails a PR which makes either worse.

### Scope (In / Out)

**In scope**

- **P0 — Baseline.** Concurrency sweep to saturation; define the SLO (TTFT and TPOT); publish a goodput curve. **Blocks every phase below.** Exit: a dated report with a goodput number and an SLO the team agrees to.
- **P1 — Concurrency.** Cap `--max-model-len` to 32768; then raise `--max-num-seqs` from 4 to the measured optimum. Exit: goodput at the new setting, measured, not assumed.
- **P2 — Cheaper tokens.** (a) FP8 weights; (b) `--kv-cache-dtype fp8_e4m3` — halves KV per token, so more concurrency at the same VRAM. **Both need a quality gate before federation** (ai-helm ADR-0118 already established fp8 KV on the 9B; it is *unverified* on this GDN hybrid). Exit: measured tok/s and a passed quality gate, or a written rejection.
- **P3 — Faster tokens.** Speculative decoding / MTP — `qwen3_next_mtp` with 2 draft tokens is already in use on the 9B. Exit: measured decode delta.
- **P4 — Two cards doing useful work.** Reclaim card 1; two replicas; prefix-affinity routing so turn *n* reaches the replica holding its KV. Exit: aggregate goodput approaching 2× a single card.
- **P5 — Hold the line.** The measurement harness is already scripted and committed; wire it as a scheduled or pre-merge gate. Exit: a deliberate regression is caught automatically.

**Out of scope**

- P/D disaggregation, wide expert parallelism, RDMA — no fabric for it.
- A network-shared KV tier — 1 GBit/s port; recompute wins ([ADR-0010](https://github.com/ADORSYS-GIS/inference-ops/pull/29)).
- Raising LMCache L1 further chasing steady-state hits — closed at ~58 GiB parity ([ADR-0012](https://github.com/ADORSYS-GIS/inference-ops/pull/29)).
- MIG / time-slicing two models onto one card — breaks the one-model-per-card rule and poisons every measurement.
- New hardware. Everything here runs on the two cards we have.
- Changing the model. That is [ADR-0006](https://github.com/ADORSYS-GIS/inference-ops/pull/29)/ADR-0008 and it needs the card P4 frees.

### Source of truth (links)

- **Every number, with its source:** https://github.com/ADORSYS-GIS/inference-ops/blob/main/docs/reference/measured-numbers.md
- What was done and the six claims it overturned: https://github.com/ADORSYS-GIS/inference-ops/blob/main/docs/explanation/2026-09-03-cache-investigation.md
- Reports: [Track 0](https://github.com/ADORSYS-GIS/inference-ops/blob/main/docs/benchmarks/2026-09-03-qwen3-5-2b-track0-cache-probe.md) · [RFC-0002 A/B](https://github.com/ADORSYS-GIS/inference-ops/blob/main/docs/benchmarks/2026-09-03-qwen3-5-2b-rfc0002-cache-ab.md)
- Roadmap and the ADRs this depends on: https://github.com/ADORSYS-GIS/inference-ops/pull/29
- Children: #1101 (restart defect), #1102 (L2 disk spike)

### Stakeholders

Fleet owner / reviewer: @Koufan-De-King. Consumers: the `adorsys-*` agent roles at the gateway, whose traffic profile (concurrent, bursty, shared system prompts) is what P0's workload must model.

### Key Assumptions

- **Decode is bandwidth-bound**, so bytes-read-per-token sets the ceiling. Backed by measurement: 280 GB/s ÷ 92 tok/s ⇒ ~2.83 GiB/token, below the 4.25 GiB of bf16 weights — consistent with the MoE FFN reading only active experts. **If this is wrong, P2(a) is worthless — verify it first.**
- FP8 weight quantization of this hybrid is supported by the engine and does not wreck quality. **Unverified. This is the single biggest assumption in the EPIC.**
- `C_hbm` stays ~995,932 tokens. It moves with `gpuMemoryUtilization` and the model, and most sizing here depends on it.
- Prefix reuse in real traffic resembles the synthetic shared-prefix workload. Unmeasured — P0 should sample real traffic rather than assume.

### Constraints

- **One model per card, one replica, `Recreate`** ⇒ each config change is a **~3.5 min production outage** of the only self-hosted model at the gateway. **Batch changes; do not merge one flag at a time.**
- In-place restarts additionally burn a failed attempt (#1101) — fix that early, it makes every later phase cheaper.
- Driver 550 / CUDA 12.4 pins engine images to `-cu129`; the default vLLM image is now CUDA 13 and needs R580+.
- Node is 62.6 GiB with Longhorn on it; the LMCache sidecar already holds ~33 GiB.
- ArgoCD polls: expect **~6–7 min** from merge to rollout start.

### Risks

- **Quantization is a quality risk, not just a speed lever.** FP8 weights and fp8 KV both change outputs. Neither ships without the production-readiness battery, and the model already has a documented repetition-loop failure mode under some sampling settings.
- **Measuring costs outages.** P0 through P4 each imply restarts. Budget them into one window rather than spreading them across a week.
- **Optimising the wrong model.** ADR-0006 is unsettled — its "25 % ceiling" premise was disproven but the dense-vs-hybrid comparison has never been run. Work invested in tuning the 2B is partly lost if the model changes. Accepted deliberately: P1–P3 are cheap and reversible.
- **A goodput SLO chosen to flatter the result.** Set it in P0, before any tuning, and do not move it afterwards.

### Success metrics

| Metric | Today | Target |
|---|---|---|
| **Goodput at a stated SLO** | **does not exist** | a published curve, then improved by a measured multiple |
| Decode, single stream | ~92 tok/s | **~180 tok/s** if the FP8 assumption holds |
| Max concurrency (KV pool ÷ max-model-len) | **3.80×** | **~30×** |
| `--max-num-seqs` | 4, unmeasured | a measured optimum |
| Prefill | ~8,600 tok/s | no regression |
| Cards serving traffic | 1 of 2 | 2 of 2 |
| Perf regression caught automatically | no | yes |

**Definition of done for the EPIC:** a dated report per phase, an ADR for every setting changed, `measured-numbers.md` updated, and a regression gate running. A phase that measures "no improvement" and says so is **done**, not failed.