[Ticket]: P1 — raise --max-num-seqs and re-run the sweep (the knee is a config constant, not the GPU)

### Type

Performance

### Summary

Raise `--max-num-seqs` on `qwen3-5-2b` and re-measure. The P0 baseline shows the fleet's throughput is capped by that one number, not by the GPU — and there is **6× of latency budget going unused** at the cap.

> 📘 **This ticket is written to be learned from, not just executed.** It explains why each step exists. Ask in the ticket if any step is unclear — an unclear step here is a bug in the ticket, not in you.

### Intent

Prove or disprove one specific claim: *`--max-num-seqs 4` is what limits this fleet, and raising it converts unused latency budget into throughput.*

### Source of truth (links)

- P0 baseline, the measurement this builds on: https://github.com/ADORSYS-GIS/inference-ops/blob/main/docs/benchmarks/2026-09-03-qwen3-5-2b-p0-baseline-sweep.md
- The SLO you are measuring against: `docs/adr/0013-serving-slo.md`
- Why the context window must be capped first: `docs/adr/0011-serve-the-small-model-at-32k.md`
- Every measured number on the fleet: `docs/reference/measured-numbers.md`
- Parent epic: #1103

### Current Behavior

From the P0 sweep — read this table before doing anything, it is the whole argument:

| Concurrency | TTFT p95 | req/s | out tok/s | SLO met |
|---|---|---|---|---|
| 4 | **0.166 s** | 1.162 | 204.9 | 100 % |
| 8 | 3.559 s | 1.164 | 201.1 | 14 % |
| 32 | 24.286 s | 1.157 | 180.5 | 8 % |

**What this says.** Throughput stops improving after concurrency 4 — 1.162, 1.164, 1.162, 1.157 req/s. It is flat. Meanwhile TTFT explodes. That combination means requests 5+ are not being *served slowly*; they are not being served at all until a slot frees. That is **admission queueing**, and `--max-num-seqs 4` is the admission limit.

The give-away is that **TPOT stays flat at ~0.040 s at every level**. Once a request is inside the batch, generation speed does not care how many others are waiting. If the GPU itself were the bottleneck, TPOT would degrade too. It doesn't — so the GPU has room.

And at concurrency 4 the p95 time-to-first-token is 0.166 s against a **1.000 s** SLO. Five sixths of the budget is unspent.

### Expected Behavior

A higher `--max-num-seqs` and a re-run sweep showing more goodput at the same SLO — or evidence that something else binds first, which is an equally good result if it is measured.

### Acceptance Criteria

- [ ] `--max-model-len` capped to **32768** first (ADR-0011). ⚠️ **Do this before raising `--max-num-seqs`.** At the current 262144 window a single request may claim a quarter of the whole KV pool, so raising the sequence limit without capping the window risks the pool running out. Capping first lifts pool-limited concurrency from **3.80× to ~30×**.
- [ ] `--max-num-seqs` raised. Suggested first step **4 → 16**; do not jump straight to 32 — a change you cannot attribute is worth very little.
- [ ] The P0 sweep re-run **unchanged** (same script, same levels, same 200 output tokens) so the two runs are comparable. Changing the workload and the config together makes the result uninterpretable.
- [ ] A new dated benchmark report in `docs/benchmarks/`, following the existing P0 report's shape, with raw evidence committed alongside.
- [ ] `docs/reference/measured-numbers.md` updated with the new goodput.
- [ ] An ADR recording the chosen `--max-num-seqs` and why.
- [ ] If goodput did **not** improve: say so plainly and write down what bound instead. A measured "no improvement" is a completed ticket, not a failed one.

### Out of Scope

- Quantization (FP8 weights, fp8 KV) — later phases of #1103; they change quality and need their own gate.
- Changing the model, the engine image, or `gpuMemoryUtilization`.
- Anything touching LMCache. It is settled for now (`docs/adr/0012-lmcache-is-restart-insurance.md`).
- Gateway-side changes.

### Technical Context

The two settings live together in `ai-helm-values`, `environments/prod/values/inference.yaml`, under the `qwen3-5-2b` entry (`serving.contextSize` and the max-num-seqs setting). **Change both in one PR** — see Constraints.

Useful background on why the KV pool matters: vLLM reserves a pool of GPU memory for the KV cache, sized by `gpuMemoryUtilization` — **not** by `--max-model-len`. On this fleet that pool is **995,932 tokens** (read it yourself from the engine's startup log; do not trust this number, verify it). `--max-model-len` decides how much of that pool one request may consume, so it sets how many requests can coexist. 995,932 ÷ 262,144 ≈ 3.8; 995,932 ÷ 32,768 ≈ 30.

Re-running the sweep:

```bash
POD=$(kubectl --context hetzner-prod -n inference get pods \
-l app.kubernetes.io/name=inference-qwen3-5-2b -o jsonpath='{.items[0].metadata.name}')

# read the pool size for yourself — this is the habit that matters
kubectl --context hetzner-prod -n inference logs "$POD" -c model | grep "GPU KV cache size"

# the same sweep P0 ran
echo '{"levels":[1,2,4,8,16,32,64],"seconds":20,"out_tokens":200,"slo_ttft":1.0,"slo_tpot":0.050}' \
| kubectl --context hetzner-prod -n inference exec -i "$POD" -c model -- \
python3 -c "$(cat docs/benchmarks/evidence/2026-09-03-p0-baseline-sweep/sweep.py)"
```

### Risks

- **This is production.** `qwen3-5-2b-local` is the only self-hosted model the gateway serves, the Deployment is `strategy: Recreate` on a single GPU, and **every config change is a ~3.5 min outage**. Batch your changes into one PR; do not merge one flag at a time.
- The sweep itself **saturates the model for ~3 minutes**. Real traffic queues behind it. Run it deliberately, not during a busy period.
- Raising `--max-num-seqs` too far trades TTFT for throughput. That is exactly the trade the SLO exists to arbitrate — if TTFT p95 crosses 1 s, you have gone too far, and that is a finding worth writing down rather than an embarrassment.
- Expect the in-place restart defect (#1101) if you restart a container by hand rather than deploying.

### Test Plan

1. Read the current pool size and `--max-num-seqs` off the running pod.
2. One PR: `contextSize` → 32768 **and** max-num-seqs → 16. Governance PR template applies.
3. Wait for the rollout (~6–7 min from merge for ArgoCD to poll, then ~3.5 min).
4. Re-read the pool size — confirm max concurrency moved from 3.80× to ~30×.
5. Re-run the sweep with the same script, adding a `64` level since the ceiling should have moved.
6. Compare against the P0 table. Report goodput, and where the new knee is.

### Verification evidence

Numbers to beat, from P0:

```text
goodput 1.162 req/s @ concurrency 4
output throughput 204.9 tok/s
TTFT p95 at knee 0.166 s (budget 1.000 s)
TPOT p95 ~0.040 s (budget 0.050 s)
knee exactly --max-num-seqs 4
```