antirez / antirez/ds4

What blocks --mtp together with --ssd-streaming? (data: it would matter most on disk-bound rigs)

未关闭
#495 9 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C
星标
22.3k
派生
2.1k
平均合并
1 天 3 小时
30 天内合并 PR
4

描述

ds4.c refuses the combination with "--ssd-streaming is not compatible with --mtp yet". I'd like to understand what the actual blocker is, because measurements on consumer hardware suggest the combination is where MTP would pay off the most.

Setup: RTX 4060 Ti 16GB, 32GB RAM, Linux/CUDA, DeepSeek V4 Flash q2 (81GB GGUF) streaming from a PCIe 3 NVMe (1.5GB/s cold reads).

Measured:
- generation: 2.2 t/s (disk-bound: ~675MB of routed experts fetched per token)
- prefill on a ~1k token prompt: 28.7 t/s, i.e. ~13x cheaper per token, because batching amortizes each expert load across the batch

MTP verification batches 2-3 tokens per forward pass, so on a disk-bound rig every accepted draft should amortize expert fetches the same way prefill does. Back of the envelope that turns 2.2 t/s into 4-6 t/s, a much bigger relative win than the same feature on machines where the model is RAM-resident.

Is the blocker that the streaming selected-expert path only handles single-token decode expert sets, or something deeper (draft/verifier KV interaction with the streaming graph)? If it's the former I'd be happy to attempt it - I have a consumer test rig (soon dual-GPU) and time to validate against the official logits fixtures.

Related: #494 (small toolkit-compat PR from the same test effort).

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。