mudler / mudler/vllm.cpp

ROCm gfx1151: no quant-matched decode number against llama.cpp on Strix Halo

Open
#2,497 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-GATE-ROCM-LLAMACPP

strix:gpu0 (gfx1151, Radeon 8060S, ROCm 7.2.4) has no quant-matched decode
number against llama.cpp, and the one control leg the board has produced is
three orders of magnitude away from what the hardware is known to give.

.agents/specs/#2377's plain_control records 0.410 tok/s for
Qwen3.8-27B bf16 (52 GiB staged) at 32 greedy tokens, qualified in its own
RESULT.json as "single unclocked control leg, not a benchmark". Published
llama.cpp figures for this exact silicon sit at 10.5-11.9 tok/s spec-off on
a ~17 GiB Q4_K GGUF:

Stack spec-off notes
Linux ROCm/HIP, ROCm 7.2.4 10.5-11.1 UD-Q4_K_XL, 96K ctx, f16 KV
Linux Vulkan/RADV 11.9 UD-Q4_K_XL, 131K ctx, q4_0 KV
Windows Vulkan 11.5 UD-Q4_K_XL, 32K ctx, q8_0 KV

Source: https://github.com/sudoingX/qwen38-mtp

The two arms are not comparable as they stand. bf16 moves ~3.2x the weight
bytes of Q4_K, and 52 GiB against a 64 GiB firmware carve leaves no room, so
residency is an uncontrolled variable on top of the quantization difference.
Neither term is measured.

This issue owns the quant-matched replacement: our Q4_K GGUF arm and the pinned
llama.cpp oracle, both on gfx1151, both on one artifact, in one lease.

Scope

  • Artifact: unsloth/Qwen3.8-27B-GGUF @ 4ca720788d1e01f1bff70c033e0d0028fd02e502,
    Qwen3.8-27B-UD-Q4_K_M.gguf, 16,464,440,224 B, sha256
    322e194ff79741c7baa497c240f677f54b201b0efab44ca8e50f122b39123482. This file
    is byte-exact with the one the published RX 7900 XTX ROCm row cites. The
    Strix Halo rows above ran UD-Q4_K_XL, whose bytes have since moved in place
    under an unchanged name (17,923,394,624 B measured, 17,559,178,144 B today),
    which is why it is not the pin here.
  • Denominator: llama.cpp b10451 (10bf611e5), the recorded oracle pin, built
    for gfx1151 in the same lease. Explicitly a secondary comparison per
    .agents/benchmarking.md.
  • Both arms: identical prompt, token count, batch 1, greedy, order-alternated
    legs, cold legs discarded for a named cause, AMD clock state sampled per leg.

The fidelity term this must carry

At b10451 llama.cpp loads 851 of this artifact's 866 tensors and ignores all
15 tensors of blk.64, four of which are the nextn.* MTP head
(.agents/oracles/llama-cpp.md). Our GGUF loader reads
qwen35.block_count = 65 with qwen35.nextn_predict_layers = 1. An arm that
runs block 64 does strictly more work per token than this denominator. The
result states which side ran what, or the ratio measures a configuration
difference and reads as a performance one.

Acceptance

  • Repeated warm-leg decode tok/s for both arms on one artifact in one lease,
    with the per-leg clock record and the boot id.
  • VT_OP_PROVIDER_STATS=1 reports zero CPU reference-tier hits on our arm.
    gfx1151 is integrated, so the reference tier is reachable and a fallback
    would otherwise be invisible in the throughput number.
  • The layer count each side ran, stated.
  • The ratio, and the named first lever if we are behind.

Related

  • #2377 owns the DFlash2 arm on this board, which hangs the GPU and is out of
    scope here.
  • #2381 owns AMD clock recording. No in-tree harness samples AMD clocks today,
    so this measurement carries an ad-hoc sampler and says so.
  • #1400, #1863, #2095, #1294, #1876, #487 are the existing ROCm decode findings
    on gfx1200. #2095 records a baseline on that board that did not reproduce
    at the same commit and file, which is why single legs are not accepted here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .agents/benchmarking.md and .agents/oracles/llama-cpp.md, then inspect the pinned b10451 llama.cpp oracle and the specified Q4_K GGUF artifact. Run both arms on gfx1151 in one lease with repeated warm legs, clock and boot records, CPU reference-tier statistics, and layer counts. Done means the ratio and first performance lever are reported with the requested fidelity details.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.