antirez / antirez/ds4

What blocks --mtp together with --ssd-streaming? (data: it would matter most on disk-bound rigs)

Aperta
#495 9 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
22.3k
Fork
2.1k
Merge medio
1g 3h
PR unite (30g)
4

Descrizione

ds4.c refuses the combination with "--ssd-streaming is not compatible with --mtp yet". I'd like to understand what the actual blocker is, because measurements on consumer hardware suggest the combination is where MTP would pay off the most.

Setup: RTX 4060 Ti 16GB, 32GB RAM, Linux/CUDA, DeepSeek V4 Flash q2 (81GB GGUF) streaming from a PCIe 3 NVMe (1.5GB/s cold reads).

Measured:
- generation: 2.2 t/s (disk-bound: ~675MB of routed experts fetched per token)
- prefill on a ~1k token prompt: 28.7 t/s, i.e. ~13x cheaper per token, because batching amortizes each expert load across the batch

MTP verification batches 2-3 tokens per forward pass, so on a disk-bound rig every accepted draft should amortize expert fetches the same way prefill does. Back of the envelope that turns 2.2 t/s into 4-6 t/s, a much bigger relative win than the same feature on machines where the model is RAM-resident.

Is the blocker that the streaming selected-expert path only handles single-token decode expert sets, or something deeper (draft/verifier KV interaction with the streaming graph)? If it's the former I'd be happy to attempt it - I have a consumer test rig (soon dual-GPU) and time to validate against the official logits fixtures.

Related: #494 (small toolkit-compat PR from the same test effort).

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.