mudler / mudler/vllm.cpp

ROCm ports zero I-quant formats, and porting a complex-vec_dot one would test whether #1910's nwarps split transfers

Open
#1,940 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT

src/vt/rocm/ ports zero I-quant formats. MatmulBTQuantKernelRocm's refusal
message at rocm_grouped_gemm.hip:495 names the gap directly:
"ported: Q8_0/Q4_K/Q5_K/Q6_K; owed: Q4_0/Q2_K/Q3_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4".
The grouped launcher's refusal at :565 repeats the same list.
.agents/specs/rocm-gg-keep-quant.md ## Boundaries already records this gap,
but no issue has tracked it until now.

What already exists to port from

CUDA and CPU both implement five I-quant formats. Porting each of these five
is an adaptation of existing source, not a from-scratch port.

Format CUDA dot Oracle
IQ2_XXS DotIQ2XXS, cuda_quant_dot.cu:307 llama.cpp quants.c:855, port note at cuda_quant_dot.cu:298
IQ3_XXS DotIQ3XXS, cuda_quant_dot.cu:340 llama.cpp quants.c:999
IQ2_S DotIQ2S, cuda_quant_dot.cu:432 llama.cpp quants.c:947, structure mirrors DotIQ3XXS
IQ1_S DotIQ1S, cuda_quant_dot.cu:383 llama.cpp quants.c:1099
IQ1_XXXS DotIQ1XXXS, cuda_quant_dot.cu:407 unslothai/llama.cpp fork quants.c:1281, registry id llama-cpp-unsloth in .agents/oracles/. This is the sub-IQ1_S encoding that a published Qwen3.8-2.4T checkpoint stores 96.92% of its routed experts in, per the port note at cuda_quant_dot.cu:710.

CPU and CUDA do not port IQ1_M, IQ2_XS, IQ3_S, IQ4_NL, or IQ4_XS
either. Those five need a fresh port from the llama.cpp oracle instead of an
adaptation.

All five already-ported formats use the same 256-element superblock and the
same BlockQ8_K activation format as Q4_K, Q5_K, and Q6_K
(kQK_K, rocm_grouped_gemm.hip:41). A ROCm port of any of them dispatches
through the identical nsb = K / 256 decomposition that
#1910 covers, including its
lane-idle defect at small nsb, if that row lands first.

Why this is more than coverage

#1910 ports llama.cpp's
nwarps-scaling decode dispatch from ggml/src/ggml-cuda/mmvq.cu, pin
10bf611e5, tag b10451. That table splits by vec_dot complexity. Q4_K,
Q5_K, and Q6_K use a linear scale and minimum, and get nwarps=8 on
RDNA4. llama.cpp excludes Q3_K and the IQ2 and IQ3 families by name,
because their vec_dot does a grid lookup. The comment at mmvq.cu:387-388
states the reason: it regresses from register pressure and lookup table
contention at higher thread counts.

We have no data on whether that split holds for our own dot product bodies.
The existing spec rocm-kquant-nwarps-decode.md measures only the simple
side. Porting IQ2_XXS or IQ3_XXS gives a complex vec_dot body to run the
same nwarps sweep against on real gfx1200 hardware. Both formats are
already adaptable from the CUDA source above, so this tests llama.cpp's
split against our own kernel bodies instead of assuming it transfers from a
different vec_dot implementation on different hardware.

What is NOT established

  • Whether the HIP translation of DotIQ2XXS or DotIQ3XXS compiles to
    efficient code on gfx1200. The grid lookup dequant can use different
    constant memory or cache behavior on RDNA4 than on the NVIDIA and CDNA
    hardware that tuned llama.cpp's table.
  • No speed or correctness claim is made here. This issue states scope and
    motivation only.
  • The keep-quant loader wiring, the dequant table adaptation from
    cpu_quant_iq_tables.h and cuda_quant_iq_tables.cuh to a ROCm
    equivalent, and the correctness tests are all unstarted.

Owned by BACKEND-ROCM.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/vt/rocm/rocm_grouped_gemm.hip, especially the refusal messages at lines 495 and 565, then compare DotIQ2XXS and DotIQ3XXS in cuda_quant_dot.cu with their listed llama.cpp oracle sources. Read the keep-quant and nwarps specs before adapting the dequant tables, loader wiring, and correctness coverage. Done means the five listed I-quant formats are supported on ROCm and IQ2/IQ3 nwarps behavior is measured on gfx1200.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai-infra-agents, backend, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.