# DSpark Speculative Decoding Slows Generation on M3 Max 128GB with the q2-imatrix (low-bit) Main Model

# DSpark Speculative Decoding Slows Generation on M3 Max 128GB with the q2-imatrix (low-bit) Main Model

**Repository:** antirez/ds4 (DwarfStar)
**Engine version:** commit `b7e9f00` — "Version DeepSeek Flash fixtures by checkpoint" (2026-08-03)
**Status:** feedback / bug report from a user benchmark

---

## 1. Summary

I benchmarked DSpark speculative decoding against plain greedy decoding on a single
Apple M3 Max 128GB machine using the **q2-imatrix** DeepSeek-V4-Flash GGUF (the 0731
checkpoint, ~91 GiB resident). In every test scenario, **enabling DSpark made generation
*slower*, not faster**, and the built-in `DS4_DSPARK_STATS=1` counters show why:
the draft model emits **no proposal in ~76–87% of cycles**, so the mean accepted
draft tokens per cycle is only **0.24–0.50** (far below the theoretical 5). The
verification + drafting overhead is therefore never amortized.

This appears to be a low-bit-quantization interaction: with an IQ2XXS-compressed main
model, the draft model's hidden-state alignment with the target is poor enough that the
scheduler almost always declines to propose. On this hardware class DSpark is a net
negative under the default configuration.

---

## 2. Environment

| Item | Value |
| --- | --- |
| Machine | MacBook Pro, Apple M3 Max, 128 GB unified memory |
| OS | macOS (Darwin 24.6.0, arm64) |
| Backend | Metal (GPU graph generation + graph prefill) |
| Engine | ds4, commit `b7e9f00` |
| Main model | `DeepSeek-V4-Flash-...-imatrix-fixed-0731.gguf` (q2-imatrix / IQ2XXS, ~91 GiB, fully resident) |
| DSpark support | `DeepSeek-V4-Flash-DSpark-support.gguf` (~5.6 GiB) |
| Launch flags | `--nothink --temp 0 -c 32768 -n 512` (greedy; required for DSpark) |

Memory plan reported by ds4 at load:
`KV 0.61 GiB + buffers 0.25 GiB + resident model 90.88 GiB = 91.74 GiB planned`
plus the ~5.6 GiB DSpark support model when enabled.

---

## 3. Reproduction

```sh
cd /path/to/ds4
MAIN=ds4flash.gguf
DSPARK=gguf/DeepSeek-V4-Flash-DSpark-support.gguf

# Baseline (greedy, no DSpark)
./ds4 -m "$MAIN" --nothink --temp 0 -c 32768 -n 512 -p "$PROMPT"

# With DSpark
./ds4 -m "$MAIN" --nothink --temp 0 -c 32768 -n 512 \
--mtp "$DSPARK" --dspark -p "$PROMPT"

# With diagnostics (acceptance counters)
DS4_DSPARK_STATS=1 ./ds4 -m "$MAIN" --nothink --temp 0 -c 32768 -n 512 \
--mtp "$DSPARK" --dspark -p "$PROMPT"
```

DSpark was confirmed enabled in the logs:

```
ds4: DSpark support model detected: .../DeepSeek-V4-Flash-DSpark-support.gguf
(stages=3 block=5 markov_rank=256 tensors=81 missing=0 invalid=0 ...)
ds4: DSpark target-hidden capture enabled: layers=40,41,42
```

The baseline run contains no `DSpark` line, confirming a clean A/B.

---

## 4. Headline results (generation tokens/sec, higher is better)

Three prompts, each run once baseline and once with DSpark (n=512, temp=0):

| Scenario | Baseline | DSpark | Delta |
| --- | --- | --- | --- |
| code — Python quicksort w/ comments | 25.91 | 25.52 | **-1.5%** |
| chat — short story (divergent) | 24.71 | 22.42 | **-9.3%** |
| struct — explain 3 parallelism strategies | 19.62 | 10.71 | **-45.4%** |
| **mean** | **~23.4** | **~19.6** | **~-16%** |

> Note on variance: in a separate diagnostic re-run (with `DS4_DSPARK_STATS=1`, same
> prompts), the DSpark generation numbers came back at **24.78** (code) and **25.00**
> (struct) — i.e. near baseline. The struct result therefore shows **high run-to-run
> variance** (10.71 vs 25.00). Even taking the more favorable re-run, DSpark provides
> no measurable speedup. The consistency of the diagnosis below is what matters.

---

## 5. DSpark diagnostics (`DS4_DSPARK_STATS=1`)

These are the raw counters printed by the engine. They explain the lack of speedup
conclusively.

### 5.1 code scenario

```
cycles=335 first_tokens=335 proposed=200 accepted_draft=168
accept_rate=84.00% avg_accept=0.501
full=64 partial=6 miss_first=9
no_draft=256 scheduler_skips=205 tail_skips=8
time_ms verify=5253.148 spec_total=5304.278 target=13873.334
saved=6779.533 net_saved=27.687
draft_len_hist=1:25,2:20,3:14,4:7,5:13
accepted_len_hist=0:265,1:24,2:21,3:9,4:5,5:11
```

### 5.2 struct scenario

```
cycles=405 first_tokens=405 proposed=114 accepted_draft=98
accept_rate=85.96% avg_accept=0.242
full=41 partial=5 miss_first=6
no_draft=353 scheduler_skips=280 tail_skips=8
time_ms verify=3092.142 spec_total=3127.028 target=16087.218
saved=3848.480 net_saved=-507.007
draft_len_hist=1:20,2:17,3:5,4:5,5:5
accepted_len_hist=0:359,1:17,2:18,3:3,4:4,5:4
```

### 5.3 What the numbers say

- **Acceptance *when a draft is proposed* is high (84–86%)** — so the verifier itself
works and accepts most proposed tokens.
- **But the draft model proposes nothing in 256/335 (76%) and 353/405 (87%) of
cycles** (`no_draft` + `scheduler_skips` dominate). The scheduler almost always
declines to draft.
- Consequently **`avg_accept` (mean accepted draft tokens per cycle) is only 0.24–0.50**,
i.e. on average DSpark advances <1 token speculatively per cycle — essentially the
same as ordinary decoding.
- The `accepted_len_hist` is dominated by `0` (265/335 and 359/405 cycles accept nothing),
confirming the scheduler bails out most of the time.
- **`net_saved` is +27.7 ms (code) and −507 ms (struct)** — both negligible-to-negative
against a ~13–16 s target decode. There is no real win to amortize the overhead.

---

## 6. Analysis / hypothesis

The draft model is a small auxiliary network that reads the target's hidden states and
proposes future tokens. Its usefulness depends on the target model's representations
being stable and predictable. With an **IQ2XXS / q2-imatrix** main model (the lowest-bit
profile the engine ships for 128 GB machines), the target's hidden states are heavily
compressed, which appears to:

1. push the scheduler into **declining to draft** in the large majority of cycles
(`no_draft` ≈ 76–87%), and
2. cap the realized acceptance length well below the 5-token block even when it does
draft (`avg_accept` ≈ 0.24–0.50).

In other words, on the lowest-bit main model, DSpark's prerequisites (a well-aligned,
predictable target) are not met, so the feature is pure overhead. This is consistent
with the README's own caveat that DSpark is "still experimental and explicitly opt-in"
and that "low-yield prompts can be no faster or even slower" — but the failure here is
not prompt-dependent; it is structural to the q2 quantization on this hardware.

It is also worth noting: the README only documents DSpark speedups qualitatively and
never gives a number even for higher-bit profiles, so there is no published baseline to
compare against for q2.

---

## 7. Questions / requests for the maintainer

1. **Is the ~76–87% `no_draft` rate expected on q2-imatrix / Metal?** If DSpark is only
intended to pay off on q4 or full-precision main models, that constraint should be
stated explicitly (today the README lists q2-imatrix as a supported DSpark host).
2. **Could the default confidence threshold (0.7, lowered from 0.9 on 2026-07-27) be
too aggressive for low-bit targets?** A lower threshold or a quantization-aware
schedule might let more drafts through and recover some benefit. A suggestion: expose
a "minimum proposal rate" guard, or auto-disable DSpark when `no_draft` stays high.
3. **Is there a known-good DSpark configuration (flag set, quantization, prompt class)
that shows a real speedup on Apple Silicon?** If so, publishing those numbers would
help users avoid the negative case.
4. The **high variance** on the struct scenario (10.71 vs 25.00 t/s across two runs with
identical flags) suggests DSpark's scheduling interacts with something
nondeterministic (thermal? memory pressure from the extra 5.6 GiB model?). Any
insight would be appreciated.

---

## 8. Files

- `run_bench.sh` — the A/B harness used for §4
- `logs/*.log` — full stdout/stderr for all six runs
- `summary.txt` — aggregated table
- (diagnostic re-runs were captured ad hoc via `DS4_DSPARK_STATS=1` and are quoted above)

Happy to re-run with different flags (e.g. `--dspark-confidence 0`, or a q4 main model
if I can free RAM / use `--ssd-streaming`) if that would help isolate the cause.