Tenstorrent keep-quant dot kernels: every GGUF k-quant arm refuses on the P150, and Qwen3.8-27B Q4_K_M is the smallest artifact that fits
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-TENSTORRENT-KEEPQUANT
What is missing
The Tenstorrent backend has no keep-quant dot kernel. DeviceKeepQuantSupported
(src/vllm/model_executor/model_loader/gguf_keep_quant.cpp:136-148) has a ROCm
arm ({Q8_0, Q4_K, Q5_K, Q6_K}) and a CUDA/CPU-fallback default; kTENSTORRENT
falls into that default, which assumes a CPU fallback the discrete P150 cannot
take, so every GGUF k-quant arm refuses on the device (docs/USAGE.md, "Arms
refused by name": "GGUF k-quant arms on TT — no TT kernels exist for them,
refused at load").
Why this row, why now
- The smallest Qwen3.8 artifact that fits the P150 is the dense 27B Q4_K_M
(17.1 GB; UD variant 16.5 GB) — the bf16 arm (53.8 GB) never fits. Without
keep-quant dots, no qwen3.8 artifact can run on the card at all. - A quantized Qwen3.5 (0.8B Q4_K_M, ~0.5 GB) is the test vehicle: the family
already adjudicates 16/16 STRICT on TT in bf16 (ambient + captured), so the
only delta between the arms is the keep-quant path itself.
Scope (this row)
- Dense keep-quant dot on TT-Metal, encodings ordered by what the vehicle and
the target artifact need: Q4_K first, then Q5_K / Q6_K / Q8_0, each gated
bit-per-block against the CPU i8mm keep-quant arm (the #2240/#2260
methodology). - The admission predicate narrowed to exactly what the new kernels implement
(gguf_keep_quant.cpp:136) — never wider than the kernel (the GLM-5.3 W10
lesson: a widened predicate throws at first forward). - Provider registration, device-resident under trace capture from day one
(#2812/#2907 discipline: no mid-capture host download, CHECK-before-download
ordering). - e2e gate: the quantized Qwen3.5 GGUF on the P150 through the existing
16-prompt battery, vs the bf16 arm's near-tie band and the CPU keep-quant
oracle.
Out of scope (owned elsewhere or follow-up)
- MoE grouped keep-quant GEMM — follow-up row once a small MoE GGUF vehicle is
pickable. - The block-decoding n-gram gather (#2394, qwen4exp-specific).
- ROCm keep-quant gaps (#2109, #1876, #1940 — ROCm lane).
- The IQ-family / sub-IQ1_S encodings — owed separately, not this row.
Correctness bar
Primary oracle: our CPU keep-quant arm (ISA-tiered; bit-exact per block vs
llama.cpp b10451). The discriminating gate is residency: the dot runs on the
TT queue with the blocks resident — on a discrete card no host fallback can
hide a missing kernel, and a dequantizing expansion that silently doubles the
bytes is a defect this row exists to prevent, not a fallback.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/vllm/model_executor/model_loader/gguf_keep_quant.cpp:136-148 and the existing CPU keep-quant arm, then review the #2240/#2260 methodology and provider registration. Use the quantized Qwen3.5 GGUF on the P150 and the existing 16-prompt battery; done means Q4_K, then the scoped encodings, run device-resident with no host fallback and match the CPU and bf16 correctness gates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100