gfx1151: publish a multi-engine throughput survey for Qwen3.8-27B Q4_K_M, with its correctness status on the face of the table
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-GATE-ROCM-LLAMACPP
Publish a multi-engine throughput survey for Qwen3.8-27B-Q4_K_M on
strix:gpu0 (gfx1151, Radeon 8060S, ROCm 7.2.4): llama.cpp b10451, the
pinned vLLM 5559679229, and vllm.cpp, on the same board, the same artifact,
the same prompt and the same token count, in one lease, as one table.
Why this exists, and what makes it admissible
The developer directed it on 2026-09-04 — "take numbers from all the engines,
and ours too, so we can compare" — after being told repeatedly that the arm's
declared token gate reads TOKEN_GATE=FAIL and that no deterministic
denominator exists on this path. The authority and its exact scope are recorded
in .agents/developer-preferences.md under "Publish a MULTI-ENGINE SURVEY on
strix:gpu0".
This is a survey, not a parity claim. #2497 already carries one retraction
for quoting a ratio while presenting it as a gated result. The only thing that
distinguishes this from that mistake is that the correctness status is published
with the numbers rather than omitted from them. So the correctness state goes
on the face of the table, not in a footnote a reader can skip:
| comparison | divergent prompts |
|---|---|
vllm.cpp vs llama.cpp b10451 |
3 of 6 |
vllm.cpp vs vLLM (torch.compile) |
5 of 6 |
vLLM (torch.compile) vs llama.cpp b10451 |
3 of 6 |
vLLM (eager) vs llama.cpp b10451 |
4 of 6 |
Every divergence is a near-tie at about 0.125 nats, one bf16 ULP. No
deterministic denominator exists on this path: llama.cpp's greedy decode is not
stable across its own kernel paths (.agents/oracles/llama-cpp.md), and vLLM
disagrees with itself across enforce_eager on both 27B models tried (#2740, and
#2915 on Qwen3.6-27B). Limb 3 of the ratified methodology is unsatisfied and may
be unsatisfiable here (#2864, #2884).
A reader must not be able to take the ratio without also taking the caveat.
Scope
- Artifact.
Qwen3.8-27B-Q4_K_M.gguf, sha256
7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b6c6fe169,
17,106,775,008 B — the same file the token gate uses, verified on the worker
before any timing runs. - llama.cpp
b10451. Already measured at 12.233 tok/s, median of 6 legs,
spread 0.303% (docs/bench-evidence/rocm-strix-llamacpp-denominator-20260902.md).
That figure is cited, and its open custody link — no compiler ran in that
lease, the binaries were carried from 2026-09-01 — is carried forward with it.
The arm is re-taken inside this survey's own lease as well, because two arms
compared across two leases are two measurements of two boxes. - vllm.cpp. New. This is what
.agents/scripts/rocm-strix-ourarm-staged.sh
was staged for; its ratification guard's release condition is now met by the
recorded developer decision. - Pinned vLLM. New. Recipe committed under
docs/bench-evidence/oracle-vllm-gfx1151-20260903/. Run in vLLM's production
configuration:AGENTS.md§Gates forbids--enforce-eageras a denominator,
so thetorch.compilearm is the headline. The correctness run's
gpu_memory_utilizationandmax_model_lenwere chosen for correctness and
are not inherited blindly.
Methodology
Matched to the landed denominator's, so the numbers are comparable: repeated warm
legs with the leg order recorded and rotated across rounds; N from the design,
never from a log tally; AMD clock windows sampled per leg; the executed kernel
path pinned as bytes, not only the revision; the artifact verified on the worker;
contention state and fleet-at-submission recorded; the arms interleaved rather
than run in blocks, so drift cannot be mistaken for a difference.
Per engine: median, mean, min/max leg, spread as a percentage of the median, and
the underlying repetition range. Then the ratios. Both absolutes always accompany
any ratio. Board faults are reported as a rate, never silently dropped.
Deliverable
One docs/benchmarks/<benchmark-id>.md detail file and its index row in
docs/BENCHMARKS.md, plus the evidence artefacts.
Related: #2497 (which this closes if it succeeds), #2534, #2864.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read .agents/developer-preferences.md and the existing denominator evidence before using .agents/scripts/rocm-strix-ourarm-staged.sh. Inspect the pinned vLLM recipe under docs/bench-evidence/oracle-vllm-gfx1151-20260903/ and reproduce the specified matched runs on strix:gpu0. Done means a table covering all three engines with correctness status, evidence artefacts, and an index row in docs/BENCHMARKS.md.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- documentation, performance
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100