antirez / antirez/ds4

Continuous depth-1 MTP speculation: +7-12% over --mtp-draft 2

Đang mở
#369 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
C
Star
22.3k
Fork
2.1k
Merge trung bình
1 ngày 3 giờ
Pull request đã merge (30 ngày)
4

Mô tả

The shipped `--mtp-draft 2` path decodes the base token on its own and then batch-verifies the MTP draft. That standalone decode is a full shared-weight pass, and the verify could have carried it. Removing it makes the MTP cycle cheaper across the board.

Instead of decoding `first_token` and then verifying the draft, draft from the trunk hidden state the previous verify already left behind, and verify `[first_token, draft]` in a single batched pass. On an accepted cycle that removes a full shared-weight pass: two tokens cost one pass instead of two. On the copy workload below that is 0.50 versus 0.70 shared reads per token, deterministic, with the base decode gone (`decodes=0`).

Measured on this branch (M4 Max, q2-q4-imatrix base, Q4K/Q8 MTP head), paired and interleaved over SSH with the GPU quiesced, round 1 discarded as warm-up:

| workload | plain AR | `--mtp-draft 2` | continuous | continuous vs draft-2 |
|---|---|---|---|---|
| code reproduction (high acceptance) | 31.2 t/s | 33.4 | 37.3 | +11.7% [11.3, 12.0] |
| technical prose | 31.4 | 32.4 | 35.9 | +10.8% [10.5, 11.0] |
| free-flowing prose (low acceptance) | 31.4 | 29.7 | 31.7 | +6.8% [6.5, 7.0] |

Continuous beats the shipped `--mtp-draft 2` by +7% to +12% across the range, because it removes the standalone base decode every cycle. Against plain autoregressive decode the gain is content-dependent, +20% on copy-heavy down to roughly flat on free prose, since the speculation benefit itself tracks draft acceptance. On free prose `--mtp-draft 2` runs about 5% slower than plain decode, and continuous stays just ahead of plain decode while still beating draft-2.

It is the same near-greedy class as the batched draft-2 verifier, not bit-exact to a strict decode. It only ever commits the verifier's argmax, so it never emits a token the target would not pick. Because it reads the base token's logits from the batched verifier rather than from sequential decode, it is a little more tie-exposed than draft-2: byte-identical to plain decode on copy-heavy output, and on prose it diverges only at genuine logit ties (worst gap 0.13 over 256 tokens here), never further. It defers to `--quality` and `DS4_MTP_STRICT`, which select the exact verifier, so callers who need bit-exactness still get it.

It is depth-1 only. It does not revive deeper drafting (the head's step-2 acceptance drops off a cliff); it makes the depth-1 cycle cheaper.

Branch: https://github.com/pandysp/ds4/tree/cont-depth1. About 120 lines, nearly all one block in `ds4_session_eval_speculative_argmax`, reusing the existing verify, draft, and prefix-1 helpers, gated behind `DS4_MTP_CONTINUOUS` and active under greedy decode with `--mtp-draft 2`, with a regression test that replays the committed stream and checks every token is the argmax. Open as PR #371; glad to have you re-measure on your own hardware.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.