antirez / antirez/ds4

DeepSeek-V4-Flash: add Q5_K/Q4_K dense-attn support (dp4a k-quant dot + fused qkv-pair/attn-out) so Q5_K-backbone GGUFs load (format support; perf caveat inside)

未关闭
#751 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C
星标
22.3k
派生
2.1k
平均合并
1 天 3 小时
30 天内合并 PR
4

描述

### Summary

ds4-server's dense-attention `matmul_quant` path only handles Q8_0/Q4_0, so serving a DeepSeek-V4-Flash GGUF whose attention/backbone tensors are Q4_K/Q5_K faults with `matmul_quant: unsupported type 12/13` at first prefill. This adds Q5_K/Q4_K support so such GGUFs load and run.

### What the patch adds

- Q5_K load acceptance in the `ds4.c` validators.
- `cuda_block_q5_K` + a `__dp4a` Q5_K×Q8_K dot (`dev_dot_q5_32`, bit-exact vs the CPU reference — packs the low nibbles + folds the 5th-bit plane 4 bytes at a time).
- Fused Q5_K kernels mirroring the Q8 fast paths: `matmul_q5_K_q8_K_pair` (shared q8_K activation quant across attn_q_a+attn_kv) and `matmul_q5_K_q8_K_hc_expand` (attn_output_b folded into the HC expansion).
- Made the decode MLA q_a/kv projections type-aware (they hardcoded the q8_0 wrapper).

Byte-identical for existing Q8 models.

### Honest perf caveat

This is **format support**, not a speed win: on GB10, a Q5_K-backbone DS4 GGUF runs at ~16.3 tok/s on ds4-server — below ds4-server's own Q8 champion (17.1) and below mainline llama.cpp on the same Q5_K weights (19.3). ds4-server's k-quant GEMV is less DRAM-efficient than the fused Q8 path, so the ~3-bit/weight saving doesn't convert to throughput at batch-1. gsm8k unchanged (dequant is bit-exact). Offering it as "can now load these GGUFs," with the caveat that Q8 remains fastest.

Patches available (naive / +dp4a / +fused). Tested on GB10 sm_121a, CUDA 13.2.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。