antirez / antirez/ds4

DeepSeek-V4-Flash: native MTP --mtp-draft>=2 corrupts output (BOS-token injection) at default (non-strict) verify — root cause + fix

Open
#750 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

### Summary

With native MTP speculative decoding at `--mtp-draft >= 2` (server defaults, i.e. no `--quality`), DeepSeek-V4-Flash output is corrupted by injected BOS tokens — e.g. `We need<|begin▁of▁sentence|># The sky is<|begin▁of▁sentence|># ...`. `--mtp-draft 1` is clean.

### Root cause (verified)

In `ds4.c` `ds4_session_eval_speculative_argmax`: without `--quality`, `strict_mtp` is false, so `use_decode2_exact` is false and draft>=2 uses the **non-exact** batched verifier `metal_graph_verify_suffix_tops` (`snapshot_required=false`) plus the no-snapshot partial-accept rollback `spec_frontier_commit_prefix1`. On a 1-of-2 partial accept, `s->logits` is set to a corrupt row-0 whose argmax is token id 0 (`<|begin▁of▁sentence|>`), so the next emitted token is BOS → self-restart cascade. Confirmed with `DS4_DSPARK_STATS`/`DS4_MTP_CONF_LOG` (`drafted=2 committed=1 target_next=0`). A correct verifier would only *reject* the bad drafts; the corruption requires the non-exact verify/rollback to wrongly *accept*.

### Fix

Route draft>=2 through the exact verifier by default (`decode2_exact` for draft==2; exact sequential fallback for draft>2 — the same path that is clean at draft=1). Verified: patched draft-1/2/4 outputs are **bit-identical** and coherent. (No throughput change — DS4 MTP gives no single-token speedup on ds4-server anyway; this is purely a correctness fix.)

### Minor

`DS4_DSPARK_STATS=1` prints per-cycle accept_rate / avg_accept / net_saved (flushed at exit or on SIGINT, which doesn't kill the server) — very useful and currently undocumented.

Patch available. Tested on GB10 (sm_121a), DeepSeek-V4-Flash-IQ2XXS chat-v2 GGUF + the MTP-Q4K drafter.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.