mudler / mudler/vllm.cpp

ROCm KQuantGemmKWmmaQ4K/Q6K prefill launches 16-48x more, smaller blocks than llama.cpp's mul_mat_q and is 4.9-10.6x slower

Open
#3,032 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: KERNEL-QUANT-CIQ-GEMM-ROCM

Row

KERNEL-QUANT-CIQ-GEMM-ROCM — already ACTIVE; this is a follow-on measurement
against the row's own landed WMMA kernels (PR #2990, #2991), not a new row.

Measured (same-tool rocprofv3, isolated pp512-only, Ornith-1.5-9B-Q4_K_M,

gfx1200 / RX 9060 XT, idle host, -r 5 for the oracle / 3 reps for ours, both
sides traced with identical flags: --kernel-trace --stats -f rocpd)

Aggregate kernel time, per rep:

Ours (KQuantGemmKWmma*) llama.cpp b10451 (mul_mat_q) Ratio
Q4_K 1,659.1 ms 156.6 ms 10.6x slower
Q6_K 323.6 ms 65.7 ms 4.9x slower
Combined 1,982.7 ms 222.3 ms 8.9x slower

Launch configuration for matching Q4_K calls (raw rocpd_kernel_dispatch
rows, same trace):

Threads/block Blocks/launch Time/launch
Ours (KQuantGemmKWmmaQ4K) 128 (4 warps) 6,144 ~19.4 ms
llama.cpp (mul_mat_q<Q4_K,128,false>) 256 (8 warps) 128–384 0.4–1.25 ms

llama.cpp uses double the warps per block and 16-48x fewer blocks, yet
finishes each launch 15-45x faster. We are not under-launching parallelism —
we launch far more blocks and still lose badly, which means each of their
blocks does dramatically more useful work per warp (more output tiles
batched per block, better data reuse) than ours does.

Why this isn't already covered by the row's existing tuning

The row's own spec (.agents/specs/kernel-quant-ciq-gemm-rocm.md, ## Now)
already swept batching within the current 4-warp block: kGroupGemmBatch
2 groups' compute+store per barrier helps (+15-32%), 4 and 8 regress
(shared-memory pressure). That tuning never touched the block's own warp
count — it is a different axis, and this measurement is the first thing
pointing at it specifically.

The work

Try an 8-warp (256-thread) variant of KQuantGemmKWmmaQ4K/
KQuantGemmKWmmaQ6K, processing more output tiles per block, matching
llama.cpp's mul_mat_q block shape (ggml/src/ggml-cuda/mmq.cuh) rather
than the current 4-warp shape. Gate it so the existing 4-warp arm stays the
default until an A/B (examples/quant-gemm-bench, same shapes as the row's
own table, best-of-N, idle host) shows a real win; record a negative result
with numbers if it does not, the same way batch=4/batch=8 were recorded
as measured-and-rejected rather than silently dropped.

Out of scope: decode (already separately measured at ~2.7x slower, unrelated
to this kernel's block shape; Q4_K's single-warp decode arm was already
measured-and-kept in ROCM-KQUANT-NWARPS-DECODE); GdnScanK and the dense
bf16 GEMM (13.3%/10.3% of prefill time respectively in this same trace,
untouched by any quant-GEMM work); gfx1100/gfx1151 (separate rows, per the
main spec's ## Owed).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the current KQuantGemmKWmmaQ4K and KQuantGemmKWmmaQ6K implementations and read .agents/specs/kernel-quant-ciq-gemm-rocm.md, then compare the block shape in ggml/src/ggml-cuda/mmq.cuh. Run examples/quant-gemm-bench using the issue's prefill shapes and benchmark conditions; done means an A/B result for the 8-warp variant, with its measured win or negative result recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.