mudler / mudler/vllm.cpp

Tenstorrent keep-quant dot kernels: every GGUF k-quant arm refuses on the P150, and Qwen3.8-27B Q4_K_M is the smallest artifact that fits

Open
#2,959 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-TENSTORRENT-KEEPQUANT

What is missing

The Tenstorrent backend has no keep-quant dot kernel. DeviceKeepQuantSupported
(src/vllm/model_executor/model_loader/gguf_keep_quant.cpp:136-148) has a ROCm
arm ({Q8_0, Q4_K, Q5_K, Q6_K}) and a CUDA/CPU-fallback default; kTENSTORRENT
falls into that default, which assumes a CPU fallback the discrete P150 cannot
take, so every GGUF k-quant arm refuses on the device (docs/USAGE.md, "Arms
refused by name": "GGUF k-quant arms on TT — no TT kernels exist for them,
refused at load").

Why this row, why now

  • The smallest Qwen3.8 artifact that fits the P150 is the dense 27B Q4_K_M
    (17.1 GB
    ; UD variant 16.5 GB) — the bf16 arm (53.8 GB) never fits. Without
    keep-quant dots, no qwen3.8 artifact can run on the card at all.
  • A quantized Qwen3.5 (0.8B Q4_K_M, ~0.5 GB) is the test vehicle: the family
    already adjudicates 16/16 STRICT on TT in bf16 (ambient + captured), so the
    only delta between the arms is the keep-quant path itself.

Scope (this row)

  • Dense keep-quant dot on TT-Metal, encodings ordered by what the vehicle and
    the target artifact need: Q4_K first, then Q5_K / Q6_K / Q8_0, each gated
    bit-per-block against the CPU i8mm keep-quant arm (the #2240/#2260
    methodology).
  • The admission predicate narrowed to exactly what the new kernels implement
    (gguf_keep_quant.cpp:136) — never wider than the kernel (the GLM-5.3 W10
    lesson: a widened predicate throws at first forward).
  • Provider registration, device-resident under trace capture from day one
    (#2812/#2907 discipline: no mid-capture host download, CHECK-before-download
    ordering).
  • e2e gate: the quantized Qwen3.5 GGUF on the P150 through the existing
    16-prompt battery, vs the bf16 arm's near-tie band and the CPU keep-quant
    oracle.

Out of scope (owned elsewhere or follow-up)

  • MoE grouped keep-quant GEMM — follow-up row once a small MoE GGUF vehicle is
    pickable.
  • The block-decoding n-gram gather (#2394, qwen4exp-specific).
  • ROCm keep-quant gaps (#2109, #1876, #1940 — ROCm lane).
  • The IQ-family / sub-IQ1_S encodings — owed separately, not this row.

Correctness bar

Primary oracle: our CPU keep-quant arm (ISA-tiered; bit-exact per block vs
llama.cpp b10451). The discriminating gate is residency: the dot runs on the
TT queue with the blocks resident — on a discrete card no host fallback can
hide a missing kernel, and a dequantizing expansion that silently doubles the
bytes is a defect this row exists to prevent, not a fallback.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/vllm/model_executor/model_loader/gguf_keep_quant.cpp:136-148 and the existing CPU keep-quant arm, then review the #2240/#2260 methodology and provider registration. Use the quantized Qwen3.5 GGUF on the P150 and the existing 16-prompt battery; done means Q4_K, then the scoped encodings, run device-resident with no host fallback and match the CPU and bf16 correctness gates.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.