ADORSYS-GIS / ADORSYS-GIS/ai-helm

[Ticket]: The 2026-08-25 direct-path benchmark measured 401s and reported a PASS — its numbers are void

Open
#1,105 0 comments 0 reactions 1 assignee Claimed by @Koufan-De-King View on GitHub
ticket
Dominant language
Go Template
Stars
3
Forks
1
Avg merge
19h 24m
Merged PRs (30d)
80

Description

### Type

Bug

### Summary

The 2026-08-25 streaming report's "direct" cells recorded **100 % HTTP 401** and were written up as a pass. Every `direct-lmcache-on` evidence file shows `codes_200: 0` — **738 rejections** across `p1`, `p3`, `p6`, `p8a`, `p8b`, with 24-byte response bodies. The report states *"direct cells now work (in-pod TTFT ~45 ms, ~90 tok/s), 0 errors ✅ PASS"*.

Those latencies are **auth-rejection times**. The harness did not send `VLLM_API_KEY`, which the model enforces.

**This matters beyond one report:** it is where the fleet's headline **"~90–94 tok/s decode"** and **"TTFT ~45 ms"** came from. Both propagated into `reference/measured-numbers.md` and were cited in ADRs and planning all through 2026-09-03.

### Intent

Replace void numbers with real ones, and stop a harness from reporting a pass on a run where nothing succeeded.

### Source of truth (links)

- Errata banner and detail: https://github.com/ADORSYS-GIS/inference-ops/blob/main/docs/benchmarks/2026-08-25-qwen3-5-2b-streaming-p8.md
- Evidence: `docs/benchmarks/evidence/2026-08-25-qwen3-5-2b-streaming-p8/direct-lmcache-on/*-events.json`
- The replacement baseline: `docs/benchmarks/2026-09-03-qwen3-5-2b-p0-baseline-sweep.md`

### Current Behavior

```
p1-direct-on-events.json codes_200 = 0 codes_401 = 30
p3-direct-on-events.json codes_200 = 0 codes_401 = 558
p6-direct-on-events.json codes_200 = 0 codes_401 = 30
p8a-direct-on-events.json codes_200 = 0 codes_401 = 60
p8b-direct-on-events.json codes_200 = 0 codes_401 = 60
```

### Expected Behavior

Direct-path cells that authenticate, return 200, and produce numbers that can be cited. And a harness that fails loudly on a non-2xx rate above the SLO instead of reporting percentiles over rejections.

### Acceptance Criteria

- [ ] The `Artillery-tests` direct scenarios send `Authorization: Bearer $VLLM_API_KEY`.
- [ ] **The harness fails the run if the non-2xx rate exceeds 1 %.** This is the real fix — a report that averages 401 latencies and prints PASS will do it again.
- [ ] The direct cells re-run and a new dated report filed. Do **not** edit the 2026-08-25 report; it is immutable and already annotated.
- [ ] `reference/measured-numbers.md` VOID rows replaced with the new figures.
- [ ] Any ADR citing ~90–94 tok/s or ~45 ms re-checked.

### Out of Scope

- The gateway cells from the same report — they returned 200 and stand.
- Re-running the whole P0–P8 battery; this is the direct path only.

### Technical Context

The model enforces `VLLM_API_KEY` (ExternalSecret `qwen3-5-2b-api-key`). In-pod, the value is already in the container's environment, which is why the `2026-09-03` driver works without ever handling the secret:

```bash
kubectl --context hetzner-prod -n inference exec -i "$POD" -c model -- python3 -c '...'
# and inside: "Authorization": "Bearer " + os.environ["VLLM_API_KEY"]
```

That pattern is worth copying — it never puts the key on a laptop.

### Risks

- Re-running the battery loads a production model; schedule it.
- The replacement numbers will likely be **worse** than the void ones (P0 measured ~64 tok/s at concurrency 1 against the claimed 90–94). Downstream capacity assumptions may need revisiting — that is the point of fixing it.

### Test Plan

1. Add auth to the direct scenarios; confirm a single request returns 200.
2. Add the non-2xx guard; confirm it fails a deliberately unauthenticated run.
3. Re-run, file a dated report, update `measured-numbers.md`.

### Verification evidence

```bash
python3 - <<'EOF'
import json,glob,re
for f in sorted(glob.glob('docs/benchmarks/evidence/2026-08-25-*/direct-lmcache-on/*-events.json')):
s=json.dumps(json.load(open(f)))
print(f.split('/')[-1], re.findall(r'"codes_(?:200|401)"\s*:\s*\d+', s))
EOF
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.