antirez / antirez/ds4

DeepSeek-V4-Flash: add Q5_K/Q4_K dense-attn support (dp4a k-quant dot + fused qkv-pair/attn-out) so Q5_K-backbone GGUFs load (format support; perf caveat inside)

オープン
#751 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
C
スター
22.3k
フォーク
2.1k
平均マージ
1日 3時間
マージ済み PR(30日)
4

説明

### Summary

ds4-server's dense-attention `matmul_quant` path only handles Q8_0/Q4_0, so serving a DeepSeek-V4-Flash GGUF whose attention/backbone tensors are Q4_K/Q5_K faults with `matmul_quant: unsupported type 12/13` at first prefill. This adds Q5_K/Q4_K support so such GGUFs load and run.

### What the patch adds

- Q5_K load acceptance in the `ds4.c` validators.
- `cuda_block_q5_K` + a `__dp4a` Q5_K×Q8_K dot (`dev_dot_q5_32`, bit-exact vs the CPU reference — packs the low nibbles + folds the 5th-bit plane 4 bytes at a time).
- Fused Q5_K kernels mirroring the Q8 fast paths: `matmul_q5_K_q8_K_pair` (shared q8_K activation quant across attn_q_a+attn_kv) and `matmul_q5_K_q8_K_hc_expand` (attn_output_b folded into the HC expansion).
- Made the decode MLA q_a/kv projections type-aware (they hardcoded the q8_0 wrapper).

Byte-identical for existing Q8 models.

### Honest perf caveat

This is **format support**, not a speed win: on GB10, a Q5_K-backbone DS4 GGUF runs at ~16.3 tok/s on ds4-server — below ds4-server's own Q8 champion (17.1) and below mainline llama.cpp on the same Q5_K weights (19.3). ds4-server's k-quant GEMV is less DRAM-efficient than the fused Q8 path, so the ~3-bit/weight saving doesn't convert to throughput at batch-1. gsm8k unchanged (dequant is bit-exact). Offering it as "can now load these GGUFs," with the caveat that Q8 remains fastest.

Patches available (naive / +dp4a / +fused). Tested on GB10 sm_121a, CUDA 13.2.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。