ROCm KQuantGemmKWmmaQ4K/Q6K prefill launches 16-48x more, smaller blocks than llama.cpp's mul_mat_q and is 4.9-10.6x slower
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: KERNEL-QUANT-CIQ-GEMM-ROCM
Row
KERNEL-QUANT-CIQ-GEMM-ROCM — already ACTIVE; this is a follow-on measurement
against the row's own landed WMMA kernels (PR #2990, #2991), not a new row.
Measured (same-tool rocprofv3, isolated pp512-only, Ornith-1.5-9B-Q4_K_M,
gfx1200 / RX 9060 XT, idle host, -r 5 for the oracle / 3 reps for ours, both
sides traced with identical flags: --kernel-trace --stats -f rocpd)
Aggregate kernel time, per rep:
Ours (KQuantGemmKWmma*) |
llama.cpp b10451 (mul_mat_q) |
Ratio | |
|---|---|---|---|
| Q4_K | 1,659.1 ms | 156.6 ms | 10.6x slower |
| Q6_K | 323.6 ms | 65.7 ms | 4.9x slower |
| Combined | 1,982.7 ms | 222.3 ms | 8.9x slower |
Launch configuration for matching Q4_K calls (raw rocpd_kernel_dispatch
rows, same trace):
| Threads/block | Blocks/launch | Time/launch | |
|---|---|---|---|
Ours (KQuantGemmKWmmaQ4K) |
128 (4 warps) | 6,144 | ~19.4 ms |
llama.cpp (mul_mat_q<Q4_K,128,false>) |
256 (8 warps) | 128–384 | 0.4–1.25 ms |
llama.cpp uses double the warps per block and 16-48x fewer blocks, yet
finishes each launch 15-45x faster. We are not under-launching parallelism —
we launch far more blocks and still lose badly, which means each of their
blocks does dramatically more useful work per warp (more output tiles
batched per block, better data reuse) than ours does.
Why this isn't already covered by the row's existing tuning
The row's own spec (.agents/specs/kernel-quant-ciq-gemm-rocm.md, ## Now)
already swept batching within the current 4-warp block: kGroupGemmBatch
2 groups' compute+store per barrier helps (+15-32%), 4 and 8 regress
(shared-memory pressure). That tuning never touched the block's own warp
count — it is a different axis, and this measurement is the first thing
pointing at it specifically.
The work
Try an 8-warp (256-thread) variant of KQuantGemmKWmmaQ4K/
KQuantGemmKWmmaQ6K, processing more output tiles per block, matching
llama.cpp's mul_mat_q block shape (ggml/src/ggml-cuda/mmq.cuh) rather
than the current 4-warp shape. Gate it so the existing 4-warp arm stays the
default until an A/B (examples/quant-gemm-bench, same shapes as the row's
own table, best-of-N, idle host) shows a real win; record a negative result
with numbers if it does not, the same way batch=4/batch=8 were recorded
as measured-and-rejected rather than silently dropped.
Out of scope: decode (already separately measured at ~2.7x slower, unrelated
to this kernel's block shape; Q4_K's single-warp decode arm was already
measured-and-kept in ROCM-KQUANT-NWARPS-DECODE); GdnScanK and the dense
bf16 GEMM (13.3%/10.3% of prefill time respectively in this same trace,
untouched by any quant-GEMM work); gfx1100/gfx1151 (separate rows, per the
main spec's ## Owed).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the current KQuantGemmKWmmaQ4K and KQuantGemmKWmmaQ6K implementations and read .agents/specs/kernel-quant-ciq-gemm-rocm.md, then compare the block shape in ggml/src/ggml-cuda/mmq.cuh. Run examples/quant-gemm-bench using the issue's prefill shapes and benchmark conditions; done means an A/B result for the 8-warp variant, with its measured win or negative result recorded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100