mudler / mudler/vllm.cpp

DotQ6K is scalar where DotQ4K uses hardware dot4, and materializes a 256-byte per-thread array; it serves ffn_down, 14.5% of Qwen3.8-27B's decode byte budget

Open
#3,017 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: KERNEL-QUANT-CIQ-GEMM-ROCM

DotQ4K uses the hardware __ockl_sdot4 wrapper (Dp4a,
c796fea41:src/vt/rocm/rocm_grouped_gemm.hip:78-88, called at :245, 64
invocations per superblock per lane). DotQ6K (:291-321) does not. It is
scalar multiply-add -- aux32[l] += scale * (q8p[l] * a[l]) at :312 and
:314 -- and it first materializes int8_t aux8[kQK_K], a 256-byte
per-thread array
(:295).

At 128 threads per block that is 32 KB of per-block private storage for one
temporary. Whether the compiler keeps it in VGPRs or spills it to scratch is
not established here and is the first thing to measure; if it spills, the arm
pays 256 bytes written plus 256 bytes read per superblock per lane on top of
the 210 bytes of Q6_K weight it is there to read, which would more than double
the traffic of the tensor it serves.

456f9cd6f added a register-resident alternative as Fmt == 3, reachable only
through VT_ROCM_Q6K_SMALL_PRIVATE, default OFF (:459-471, :741-762). It
was added as a diagnostic A/B for the gfx1151 hang in #2511 and no throughput
verdict for it is recorded.

What it costs on the survey workload

On Qwen3.8-27B-Q4_K_M, ffn_down is the only projection with K = 17408, so
nsb = 68 > 32 and the cooperative Q6_K arm never fires for it
(KQuantDecodeCoopWarps, :777-788). Its 32 Q6_K layers are 14.54% of the
per-token swept byte budget
(2,339,635,200 of 16,091,088,896 B) and they run
KQuantGemmK<OutT,2> with the scalar DotQ6K body. Lane occupancy is fine
here -- nsb = 68 fills all 32 lanes -- so this is a distinct defect from
#3016, on a distinct set of bytes, and the two do not overlap.

The ffn_down Q4_K half (32 layers, 9.97% of the budget) takes the same kernel
with the dp4a DotQ4K body, which makes this an A/B that already exists in the
same model: two arms of one tensor role, identical shape and lane occupancy,
differing only in the inner product. A rocprofv3 trace (#3015) separates them
by template instantiation without any new harness.

What is asked

  1. Measure whether aux8[256] spills, by reading the ISA
    (--save-temps or llvm-objdump for ScratchSize on the
    KQuantGemmK<*, 2> instantiation). This needs no GPU.
  2. Give DotQ6K the same Dp4a treatment DotQ4K has, or promote Fmt == 3
    if it measures ahead, and record the verdict the diagnostic knob never got.

Needs a spec before code.

Related: #3016, #3015, #2109, #2511, #1910, #41.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/vt/rocm/rocm_grouped_gemm.hip:291-321 and compare DotQ6K with DotQ4K at :78-88 and :245. Inspect the KQuantGemmK<*, 2> ISA using --save-temps or llvm-objdump, especially ScratchSize, then compare the existing Fmt == 3 path at :459-471. Done when spill status, the rocprofv3 performance verdict, and the resulting specification are recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.