antirez / antirez/ds4

What blocks --mtp together with --ssd-streaming? (data: it would matter most on disk-bound rigs)

Abierto
#495 9 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.3k
Forks
2.1k
Merge medio
1 d 3 h
PR fusionados (30 d)
4

Descripción

ds4.c refuses the combination with "--ssd-streaming is not compatible with --mtp yet". I'd like to understand what the actual blocker is, because measurements on consumer hardware suggest the combination is where MTP would pay off the most.

Setup: RTX 4060 Ti 16GB, 32GB RAM, Linux/CUDA, DeepSeek V4 Flash q2 (81GB GGUF) streaming from a PCIe 3 NVMe (1.5GB/s cold reads).

Measured:
- generation: 2.2 t/s (disk-bound: ~675MB of routed experts fetched per token)
- prefill on a ~1k token prompt: 28.7 t/s, i.e. ~13x cheaper per token, because batching amortizes each expert load across the batch

MTP verification batches 2-3 tokens per forward pass, so on a disk-bound rig every accepted draft should amortize expert fetches the same way prefill does. Back of the envelope that turns 2.2 t/s into 4-6 t/s, a much bigger relative win than the same feature on machines where the model is RAM-resident.

Is the blocker that the streaming selected-expert path only handles single-token decode expert sets, or something deeper (draft/verifier KV interaction with the streaming graph)? If it's the former I'd be happy to attempt it - I have a consumer test rig (soon dual-GPU) and time to validate against the official logits fixtures.

Related: #494 (small toolkit-compat PR from the same test effort).

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.