DeepSeek-V4-Flash: add Q5_K/Q4_K dense-attn support (dp4a k-quant dot + fused qkv-pair/attn-out) so Q5_K-backbone GGUFs load (format support; perf caveat inside)
- 主要言語
- C
- スター
- 22.3k
- フォーク
- 2.1k
- 平均マージ
- 1日 3時間
- マージ済み PR(30日)
- 4
説明
### Summary
ds4-server's dense-attention `matmul_quant` path only handles Q8_0/Q4_0, so serving a DeepSeek-V4-Flash GGUF whose attention/backbone tensors are Q4_K/Q5_K faults with `matmul_quant: unsupported type 12/13` at first prefill. This adds Q5_K/Q4_K support so such GGUFs load and run.
### What the patch adds
- Q5_K load acceptance in the `ds4.c` validators.
- `cuda_block_q5_K` + a `__dp4a` Q5_K×Q8_K dot (`dev_dot_q5_32`, bit-exact vs the CPU reference — packs the low nibbles + folds the 5th-bit plane 4 bytes at a time).
- Fused Q5_K kernels mirroring the Q8 fast paths: `matmul_q5_K_q8_K_pair` (shared q8_K activation quant across attn_q_a+attn_kv) and `matmul_q5_K_q8_K_hc_expand` (attn_output_b folded into the HC expansion).
- Made the decode MLA q_a/kv projections type-aware (they hardcoded the q8_0 wrapper).
Byte-identical for existing Q8 models.
### Honest perf caveat
This is **format support**, not a speed win: on GB10, a Q5_K-backbone DS4 GGUF runs at ~16.3 tok/s on ds4-server — below ds4-server's own Q8 champion (17.1) and below mainline llama.cpp on the same Q5_K weights (19.3). ds4-server's k-quant GEMV is less DRAM-efficient than the fused Q8 path, so the ~3-bit/weight saving doesn't convert to throughput at batch-1. gsm8k unchanged (dequant is bit-exact). Offering it as "can now load these GGUFs," with the caveat that Q8 remains fastest.
Patches available (naive / +dp4a / +fused). Tested on GB10 sm_121a, CUDA 13.2.
コントリビューションガイド
評価
この issue はまだ評価されていません。