antirez / antirez/ds4

What blocks --mtp together with --ssd-streaming? (data: it would matter most on disk-bound rigs)

オープン
#495 コメント 9 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
C
スター
22.3k
フォーク
2.1k
平均マージ
1日 3時間
マージ済み PR(30日)
4

説明

ds4.c refuses the combination with "--ssd-streaming is not compatible with --mtp yet". I'd like to understand what the actual blocker is, because measurements on consumer hardware suggest the combination is where MTP would pay off the most.

Setup: RTX 4060 Ti 16GB, 32GB RAM, Linux/CUDA, DeepSeek V4 Flash q2 (81GB GGUF) streaming from a PCIe 3 NVMe (1.5GB/s cold reads).

Measured:
- generation: 2.2 t/s (disk-bound: ~675MB of routed experts fetched per token)
- prefill on a ~1k token prompt: 28.7 t/s, i.e. ~13x cheaper per token, because batching amortizes each expert load across the batch

MTP verification batches 2-3 tokens per forward pass, so on a disk-bound rig every accepted draft should amortize expert fetches the same way prefill does. Back of the envelope that turns 2.2 t/s into 4-6 t/s, a much bigger relative win than the same feature on machines where the model is RAM-resident.

Is the blocker that the streaming selected-expert path only handles single-token decode expert sets, or something deeper (draft/verifier KV interaction with the streaming graph)? If it's the former I'd be happy to attempt it - I have a consumer test rig (soon dual-GPU) and time to validate against the official logits fixtures.

Related: #494 (small toolkit-compat PR from the same test effort).

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。