antirez / antirez/ds4

Speculative Decoding via "Draft KV Stitching"

Open
#63 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

## Description
The current MTP (Multi-Token Prediction) path is experimental and provides "at most a slight speedup." We can improve this by implementing Draft KV Stitching.

## Proposed Changes
* **Small Draft Model Integration:** Allow the engine to load a tiny (e.g., DeepSeek 1.3B) model purely for drafting.
* **Speculative Verification:** The main DS4 engine verifies 4-8 draft tokens in a single Metal graph pass.
* **KV Alignment:** Ensure that when a draft is accepted, the "compressed KV" of the main model is updated without a full re-prefill.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.