antirez / antirez/ds4

DeepSeek-V4-Flash: add Q5_K/Q4_K dense-attn support (dp4a k-quant dot + fused qkv-pair/attn-out) so Q5_K-backbone GGUFs load (format support; perf caveat inside)

Open
#751 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

### Summary

ds4-server's dense-attention `matmul_quant` path only handles Q8_0/Q4_0, so serving a DeepSeek-V4-Flash GGUF whose attention/backbone tensors are Q4_K/Q5_K faults with `matmul_quant: unsupported type 12/13` at first prefill. This adds Q5_K/Q4_K support so such GGUFs load and run.

### What the patch adds

- Q5_K load acceptance in the `ds4.c` validators.
- `cuda_block_q5_K` + a `__dp4a` Q5_K×Q8_K dot (`dev_dot_q5_32`, bit-exact vs the CPU reference — packs the low nibbles + folds the 5th-bit plane 4 bytes at a time).
- Fused Q5_K kernels mirroring the Q8 fast paths: `matmul_q5_K_q8_K_pair` (shared q8_K activation quant across attn_q_a+attn_kv) and `matmul_q5_K_q8_K_hc_expand` (attn_output_b folded into the HC expansion).
- Made the decode MLA q_a/kv projections type-aware (they hardcoded the q8_0 wrapper).

Byte-identical for existing Q8 models.

### Honest perf caveat

This is **format support**, not a speed win: on GB10, a Q5_K-backbone DS4 GGUF runs at ~16.3 tok/s on ds4-server — below ds4-server's own Q8 champion (17.1) and below mainline llama.cpp on the same Q5_K weights (19.3). ds4-server's k-quant GEMV is less DRAM-efficient than the fused Q8 path, so the ~3-bit/weight saving doesn't convert to throughput at batch-1. gsm8k unchanged (dequant is bit-exact). Offering it as "can now load these GGUFs," with the caveat that Q8 remains fastest.

Patches available (naive / +dp4a / +fused). Tested on GB10 sm_121a, CUDA 13.2.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.