antirez / antirez/ds4

Continuous depth-1 MTP speculation: +7-12% over --mtp-draft 2

Abierto
#369 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.3k
Forks
2.1k
Merge medio
1 d 3 h
PR fusionados (30 d)
4

Descripción

The shipped `--mtp-draft 2` path decodes the base token on its own and then batch-verifies the MTP draft. That standalone decode is a full shared-weight pass, and the verify could have carried it. Removing it makes the MTP cycle cheaper across the board.

Instead of decoding `first_token` and then verifying the draft, draft from the trunk hidden state the previous verify already left behind, and verify `[first_token, draft]` in a single batched pass. On an accepted cycle that removes a full shared-weight pass: two tokens cost one pass instead of two. On the copy workload below that is 0.50 versus 0.70 shared reads per token, deterministic, with the base decode gone (`decodes=0`).

Measured on this branch (M4 Max, q2-q4-imatrix base, Q4K/Q8 MTP head), paired and interleaved over SSH with the GPU quiesced, round 1 discarded as warm-up:

| workload | plain AR | `--mtp-draft 2` | continuous | continuous vs draft-2 |
|---|---|---|---|---|
| code reproduction (high acceptance) | 31.2 t/s | 33.4 | 37.3 | +11.7% [11.3, 12.0] |
| technical prose | 31.4 | 32.4 | 35.9 | +10.8% [10.5, 11.0] |
| free-flowing prose (low acceptance) | 31.4 | 29.7 | 31.7 | +6.8% [6.5, 7.0] |

Continuous beats the shipped `--mtp-draft 2` by +7% to +12% across the range, because it removes the standalone base decode every cycle. Against plain autoregressive decode the gain is content-dependent, +20% on copy-heavy down to roughly flat on free prose, since the speculation benefit itself tracks draft acceptance. On free prose `--mtp-draft 2` runs about 5% slower than plain decode, and continuous stays just ahead of plain decode while still beating draft-2.

It is the same near-greedy class as the batched draft-2 verifier, not bit-exact to a strict decode. It only ever commits the verifier's argmax, so it never emits a token the target would not pick. Because it reads the base token's logits from the batched verifier rather than from sequential decode, it is a little more tie-exposed than draft-2: byte-identical to plain decode on copy-heavy output, and on prose it diverges only at genuine logit ties (worst gap 0.13 over 256 tokens here), never further. It defers to `--quality` and `DS4_MTP_STRICT`, which select the exact verifier, so callers who need bit-exactness still get it.

It is depth-1 only. It does not revive deeper drafting (the head's step-2 acceptance drops off a cliff); it makes the depth-1 cycle cheaper.

Branch: https://github.com/pandysp/ds4/tree/cont-depth1. About 120 lines, nearly all one block in `ds4_session_eval_speculative_argmax`, reusing the existing verify, draft, and prefix-1 helpers, gated behind `DS4_MTP_CONTINUOUS` and active under greedy decode with `--mtp-draft 2`, with a regression test that replays the committed stream and checks every token is the argmax. Open as PR #371; glad to have you re-measure on your own hardware.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.