antirez / antirez/ds4

DeepSeek-V4-Flash: add Q5_K/Q4_K dense-attn support (dp4a k-quant dot + fused qkv-pair/attn-out) so Q5_K-backbone GGUFs load (format support; perf caveat inside)

Đang mở
#751 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
C
Star
22.3k
Fork
2.1k
Merge trung bình
1 ngày 3 giờ
Pull request đã merge (30 ngày)
4

Mô tả

### Summary

ds4-server's dense-attention `matmul_quant` path only handles Q8_0/Q4_0, so serving a DeepSeek-V4-Flash GGUF whose attention/backbone tensors are Q4_K/Q5_K faults with `matmul_quant: unsupported type 12/13` at first prefill. This adds Q5_K/Q4_K support so such GGUFs load and run.

### What the patch adds

- Q5_K load acceptance in the `ds4.c` validators.
- `cuda_block_q5_K` + a `__dp4a` Q5_K×Q8_K dot (`dev_dot_q5_32`, bit-exact vs the CPU reference — packs the low nibbles + folds the 5th-bit plane 4 bytes at a time).
- Fused Q5_K kernels mirroring the Q8 fast paths: `matmul_q5_K_q8_K_pair` (shared q8_K activation quant across attn_q_a+attn_kv) and `matmul_q5_K_q8_K_hc_expand` (attn_output_b folded into the HC expansion).
- Made the decode MLA q_a/kv projections type-aware (they hardcoded the q8_0 wrapper).

Byte-identical for existing Q8 models.

### Honest perf caveat

This is **format support**, not a speed win: on GB10, a Q5_K-backbone DS4 GGUF runs at ~16.3 tok/s on ds4-server — below ds4-server's own Q8 champion (17.1) and below mainline llama.cpp on the same Q5_K weights (19.3). ds4-server's k-quant GEMV is less DRAM-efficient than the fused Q8 path, so the ~3-bit/weight saving doesn't convert to throughput at batch-1. gsm8k unchanged (dequant is bit-exact). Offering it as "can now load these GGUFs," with the caveat that Q8 remains fastest.

Patches available (naive / +dp4a / +fused). Tested on GB10 sm_121a, CUDA 13.2.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.