mudler / mudler/vllm.cpp

gfx1151: publish a multi-engine throughput survey for Qwen3.8-27B Q4_K_M, with its correctness status on the face of the table

Open
#2,921 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-GATE-ROCM-LLAMACPP

Publish a multi-engine throughput survey for Qwen3.8-27B-Q4_K_M on
strix:gpu0 (gfx1151, Radeon 8060S, ROCm 7.2.4): llama.cpp b10451, the
pinned vLLM 5559679229, and vllm.cpp, on the same board, the same artifact,
the same prompt and the same token count, in one lease, as one table.

Why this exists, and what makes it admissible

The developer directed it on 2026-09-04 — "take numbers from all the engines,
and ours too, so we can compare"
— after being told repeatedly that the arm's
declared token gate reads TOKEN_GATE=FAIL and that no deterministic
denominator exists on this path. The authority and its exact scope are recorded
in .agents/developer-preferences.md under "Publish a MULTI-ENGINE SURVEY on
strix:gpu0"
.

This is a survey, not a parity claim. #2497 already carries one retraction
for quoting a ratio while presenting it as a gated result. The only thing that
distinguishes this from that mistake is that the correctness status is published
with the numbers rather than omitted from them. So the correctness state goes
on the face of the table, not in a footnote a reader can skip:

comparison divergent prompts
vllm.cpp vs llama.cpp b10451 3 of 6
vllm.cpp vs vLLM (torch.compile) 5 of 6
vLLM (torch.compile) vs llama.cpp b10451 3 of 6
vLLM (eager) vs llama.cpp b10451 4 of 6

Every divergence is a near-tie at about 0.125 nats, one bf16 ULP. No
deterministic denominator exists on this path
: llama.cpp's greedy decode is not
stable across its own kernel paths (.agents/oracles/llama-cpp.md), and vLLM
disagrees with itself across enforce_eager on both 27B models tried (#2740, and
#2915 on Qwen3.6-27B). Limb 3 of the ratified methodology is unsatisfied and may
be unsatisfiable here (#2864, #2884).

A reader must not be able to take the ratio without also taking the caveat.

Scope

  • Artifact. Qwen3.8-27B-Q4_K_M.gguf, sha256
    7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b6c6fe169,
    17,106,775,008 B — the same file the token gate uses, verified on the worker
    before any timing runs.
  • llama.cpp b10451. Already measured at 12.233 tok/s, median of 6 legs,
    spread 0.303% (docs/bench-evidence/rocm-strix-llamacpp-denominator-20260902.md).
    That figure is cited, and its open custody link — no compiler ran in that
    lease, the binaries were carried from 2026-09-01 — is carried forward with it.
    The arm is re-taken inside this survey's own lease as well, because two arms
    compared across two leases are two measurements of two boxes.
  • vllm.cpp. New. This is what .agents/scripts/rocm-strix-ourarm-staged.sh
    was staged for; its ratification guard's release condition is now met by the
    recorded developer decision.
  • Pinned vLLM. New. Recipe committed under
    docs/bench-evidence/oracle-vllm-gfx1151-20260903/. Run in vLLM's production
    configuration
    : AGENTS.md §Gates forbids --enforce-eager as a denominator,
    so the torch.compile arm is the headline. The correctness run's
    gpu_memory_utilization and max_model_len were chosen for correctness and
    are not inherited blindly.

Methodology

Matched to the landed denominator's, so the numbers are comparable: repeated warm
legs with the leg order recorded and rotated across rounds; N from the design,
never from a log tally
; AMD clock windows sampled per leg; the executed kernel
path pinned as bytes, not only the revision; the artifact verified on the worker;
contention state and fleet-at-submission recorded; the arms interleaved rather
than run in blocks, so drift cannot be mistaken for a difference.

Per engine: median, mean, min/max leg, spread as a percentage of the median, and
the underlying repetition range. Then the ratios. Both absolutes always accompany
any ratio. Board faults are reported as a rate, never silently dropped.

Deliverable

One docs/benchmarks/<benchmark-id>.md detail file and its index row in
docs/BENCHMARKS.md, plus the evidence artefacts.

Related: #2497 (which this closes if it succeeds), #2534, #2864.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read .agents/developer-preferences.md and the existing denominator evidence before using .agents/scripts/rocm-strix-ourarm-staged.sh. Inspect the pinned vLLM recipe under docs/bench-evidence/oracle-vllm-gfx1151-20260903/ and reproduce the specified matched runs on strix:gpu0. Done means a table covering all three engines with correctness status, evidence artefacts, and an index row in docs/BENCHMARKS.md.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
documentation, performance
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.