antirez / antirez/ds4

Speculative Decoding via "Draft KV Stitching"

未关闭
#63 1 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C
星标
22.3k
派生
2.1k
平均合并
1 天 3 小时
30 天内合并 PR
4

描述

## Description
The current MTP (Multi-Token Prediction) path is experimental and provides "at most a slight speedup." We can improve this by implementing Draft KV Stitching.

## Proposed Changes
* **Small Draft Model Integration:** Allow the engine to load a tiny (e.g., DeepSeek 1.3B) model purely for drafting.
* **Speculative Verification:** The main DS4 engine verifies 4-8 draft tokens in a single Metal graph pass.
* **KV Alignment:** Ensure that when a draft is accepted, the "compressed KV" of the main model is updated without a full re-prefill.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。