antirez / antirez/ds4

Speculative Decoding via "Draft KV Stitching"

Abierto
#63 1 comentario 1 reacción 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.3k
Forks
2.1k
Merge medio
1 d 3 h
PR fusionados (30 d)
4

Descripción

## Description
The current MTP (Multi-Token Prediction) path is experimental and provides "at most a slight speedup." We can improve this by implementing Draft KV Stitching.

## Proposed Changes
* **Small Draft Model Integration:** Allow the engine to load a tiny (e.g., DeepSeek 1.3B) model purely for drafting.
* **Speculative Verification:** The main DS4 engine verifies 4-8 draft tokens in a single Metal graph pass.
* **KV Alignment:** Ensure that when a draft is accepted, the "compressed KV" of the main model is updated without a full re-prefill.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.