mudler / mudler/vllm.cpp

gfx1151 falls back to the legacy QuantizeQ8KK: the cooperative Q8_K activation quantizer is gated on a strict gfx1100 prefix

Open
#3,018 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM

#1876 measured QuantizeQ8KK, the thread-per-superblock activation quantizer,
at 95.5 us/call against llama.cpp's 1.6 us, and put it at 41% of the ROCm
decode gap on gfx1200. 70d8d317e and 17f4568d1 landed the fix, a
block-per-superblock cooperative quantizer with a shared-memory argmax
reduction (QuantizeQ8KCooperativeK,
c796fea41:src/vt/rocm/rocm_grouped_gemm.hip:153-208).

gfx1151 does not get it. The selector is gated on a strict gfx1100 prefix
test (:641-643, :673-694):

constexpr bool kQ8KGfx1100DefaultAccepted = true;
...
return gfx1100_default_accepted && Q8KArchIsGfx1100(arch)
           ? Q8KQuantArm::kCandidate
           : Q8KQuantArm::kLegacy;

So on Strix Halo every MatmulBTQuant dispatch quantizes its activation with
QuantizeQ8KK at 128 threads (:714-717). VT_ROCM_Q8K_BLOCK=1 overrides it.

The gate is scoped that narrowly on purpose -- the comment at :641-643 says
the real-checkpoint and profiler evidence covered gfx1100 only, which is the
right discipline. This issue is the missing measurement, not a complaint about
the gate.

Why it is worth measuring rather than assuming

At K = 5120 the activation is one row of nsb = 20 superblocks, so the
legacy kernel launches one block with 20 of 128 threads doing work, each
serially reducing 256 elements. The kernel is tiny in bytes and large in
latency, and it runs once per quantized matmul: on Qwen3.8-27B that is
roughly five to seven dispatches per layer across 64 layers, so a few hundred
launches per token.

That is a fixed per-dispatch cost, which is why it was 41% of a 9B model's gap
and is expected to be a much smaller share of a 27B one -- 27B decode is
185.3 ms/token (#3002) against the 9B's 57.3 ms, over similar dispatch counts.
No figure is claimed here. The point is that the cost is unmeasured on this
board and the cheap arm is already written and gated off.

What is asked

VT_ROCM_Q8K_BLOCK=1 versus default, same binary, same lease, on
Qwen3.8-27B-Q4_K_M on strix:gpu0. If it wins, widen Q8KArchIsGfx1100 to
carry gfx1151 with the evidence beside it; if it loses, record that on this
row so the next reader does not re-derive it. Either result can ride #3015's
profiling lease.

Related: #1876, #3015, #3016, #2497, #2921, #41.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/vt/rocm/rocm_grouped_gemm.hip at the selector around lines 641-717, then compare VT_ROCM_Q8K_BLOCK=1 with the default on the specified strix:gpu0 lease using Qwen3.8-27B-Q4_K_M and the same binary. Done means recording the benchmark result on this issue and, only if it wins, updating the gfx1151 gate with the supporting evidence.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.