Continuous depth-1 MTP speculation: +7-12% over --mtp-draft 2
- Ngôn ngữ chính
- C
- Star
- 22.3k
- Fork
- 2.1k
- Merge trung bình
- 1 ngày 3 giờ
- Pull request đã merge (30 ngày)
- 4
Mô tả
The shipped `--mtp-draft 2` path decodes the base token on its own and then batch-verifies the MTP draft. That standalone decode is a full shared-weight pass, and the verify could have carried it. Removing it makes the MTP cycle cheaper across the board.
Instead of decoding `first_token` and then verifying the draft, draft from the trunk hidden state the previous verify already left behind, and verify `[first_token, draft]` in a single batched pass. On an accepted cycle that removes a full shared-weight pass: two tokens cost one pass instead of two. On the copy workload below that is 0.50 versus 0.70 shared reads per token, deterministic, with the base decode gone (`decodes=0`).
Measured on this branch (M4 Max, q2-q4-imatrix base, Q4K/Q8 MTP head), paired and interleaved over SSH with the GPU quiesced, round 1 discarded as warm-up:
| workload | plain AR | `--mtp-draft 2` | continuous | continuous vs draft-2 |
|---|---|---|---|---|
| code reproduction (high acceptance) | 31.2 t/s | 33.4 | 37.3 | +11.7% [11.3, 12.0] |
| technical prose | 31.4 | 32.4 | 35.9 | +10.8% [10.5, 11.0] |
| free-flowing prose (low acceptance) | 31.4 | 29.7 | 31.7 | +6.8% [6.5, 7.0] |
Continuous beats the shipped `--mtp-draft 2` by +7% to +12% across the range, because it removes the standalone base decode every cycle. Against plain autoregressive decode the gain is content-dependent, +20% on copy-heavy down to roughly flat on free prose, since the speculation benefit itself tracks draft acceptance. On free prose `--mtp-draft 2` runs about 5% slower than plain decode, and continuous stays just ahead of plain decode while still beating draft-2.
It is the same near-greedy class as the batched draft-2 verifier, not bit-exact to a strict decode. It only ever commits the verifier's argmax, so it never emits a token the target would not pick. Because it reads the base token's logits from the batched verifier rather than from sequential decode, it is a little more tie-exposed than draft-2: byte-identical to plain decode on copy-heavy output, and on prose it diverges only at genuine logit ties (worst gap 0.13 over 256 tokens here), never further. It defers to `--quality` and `DS4_MTP_STRICT`, which select the exact verifier, so callers who need bit-exactness still get it.
It is depth-1 only. It does not revive deeper drafting (the head's step-2 acceptance drops off a cliff); it makes the depth-1 cycle cheaper.
Branch: https://github.com/pandysp/ds4/tree/cont-depth1. About 120 lines, nearly all one block in `ds4_session_eval_speculative_argmax`, reusing the existing verify, draft, and prefix-1 helpers, gated behind `DS4_MTP_CONTINUOUS` and active under greedy decode with `--mtp-draft 2`, with a regression test that replays the committed stream and checks every token is the argmax. Open as PR #371; glad to have you re-measure on your own hardware.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.