antirez / antirez/ds4

DSpark net-negative on M3 Ultra + MXFP4 despite 82.8% accept rate — independent confirmation of #695 replay/verify overhead

Open
#733 9 comments 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

## Summary

Adding a cross-configuration data point for #695: DSpark is net-negative on **M3 Ultra 256GB + MXFP4** (the recommended quantization), even though the draft acceptance rate is **82.8%**. This confirms the core finding of #695 from a different hardware generation and quantization format: the bottleneck is replay + verify overhead, not draft quality.

## Environment

| Item | Value |
|------|-------|
| Chip | Apple M3 Ultra (256 GB unified memory, 819 GB/s) |
| Engine | ds4 commit `b030961` |
| Model GGUF | `DeepSeek-V4-Flash-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-mxfp4-0731.gguf` (145 GB) |
| DSpark support | `DeepSeek-V4-Flash-DSpark-support-0731.gguf` (5.6 GB) |
| Quantization | MXFP4 experts (native, bit-for-bit) + F16/Q8 non-expert |
| Context | 8192, `--nothink --temp 0` |
| Prompt | `"Write a Python quicksort function with comments."` (100 tokens) |

## Controlled Experiment (4 modes, same prompt)

| Mode | Generation (t/s) | Decode vs baseline |
|------|-----------------|-------------------|
| Baseline (no DSpark) | 39.78 | — |
| `--dspark-strict` (loaded, target-only) | 39.17 | -1.5% (loading overhead ≈ 0) |
| `--dspark` (confidence 0.7, default) | 25.47 | **-36%** |
| `--dspark --dspark-confidence 0` (forced 5-token blocks) | 21.58 | **-46%** |

The strict experiment isolates the overhead: loading the DSpark support model costs ~1.5% (within noise). The regression is entirely in the runtime decode path.

## DSpark Telemetry (`DS4_DSPARK_STATS=1`)

### Default confidence (0.7)

```
cycles=46 proposed=58 accepted_draft=48 accept_rate=82.76% avg_accept=1.043
full=12 partial=4 miss_first=2 no_draft=28 scheduler_skips=23
time_ms propose=379.8 verify=1236.4 replay=1206.8 spec_total=2460.6
target=1571.3 saved=1631.7 net_saved=-1208.7
draft_len_hist=1:2,2:4,3:5,4:2,5:5
accepted_len_hist=0:30,1:3,2:3,3:5,4:1,5:4
```

### Forced 5-token blocks (confidence 0)

```
cycles=24 proposed=120 accepted_draft=70 accept_rate=58.33% avg_accept=2.917
full=7 partial=16 miss_first=1 no_draft=0 scheduler_skips=0
time_ms propose=221.2 verify=1928.3 replay=1746.3 spec_total=3698.1
target=760.6 saved=1794.5 net_saved=-2124.8
draft_len_hist=5:24
accepted_len_hist=0:1,1:5,2:6,3:2,4:3,5:7
```

## Analysis: Where the Loss Comes From

All three time components in this run are in the same order of magnitude:

| Component | Time (ms) | Per-cycle (ms) | Notes |
|-----------|----------|----------------|-------|
| propose | 379.8 | 8.3 | Draft generation (cheap) |
| verify | 1236.4 | 26.9 | Target model verification |
| replay | 1206.8 | 26.2 | Partial-accept replay (`7fb2830`) |

**replay (1207 ms) ≈ verify (1236 ms)**. The replay cost — introduced by commit `7fb2830` ("Replay partial DSpark accepts through ordinary decode") for correctness — is as expensive as the verify pass itself. Combined, replay + verify = 2443 ms, while accepted tokens only save 1632 ms.

This is consistent with the finding in #695: the scheduler's break-even model may not account for replay cost, preventing convergence to baseline even when draft quality is good.

## What This Confirms (and What It Doesn't)

**Confirms**: Draft quality is not the bottleneck on MXFP4. An 82.8% acceptance rate — higher than expected — still yields a net loss. The regression is driven by replay + verify overhead.

**Does not claim**: This report does not compare absolute timing against #695, as the test scales differ (#695 covers a multi-thousand-token corpus; this is a 100-token single prompt). Per-cycle costs are not directly comparable across different draft block sizes and corpus lengths. The shared conclusion is qualitative: both reports independently observe net-negative results despite healthy acceptance rates.

## Scheduler Behavior

In default mode (confidence 0.7), the scheduler skips 23 of 46 cycles (`no_draft=28`). This is the correct behavior — when forced to propose every cycle (confidence 0), net loss worsens from -1209 ms to -2125 ms. The scheduler is already doing damage control, but as #695 notes, the break-even model may not fully account for replay cost, preventing convergence to baseline.

## Repro

```bash
# Baseline
./ds4 -m --ctx 8192 --nothink --temp 0 \
-p "Write a Python quicksort function with comments." --tokens 100

# DSpark with telemetry
DS4_DSPARK_STATS=1 ./ds4 \
-m \
--mtp gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \
--dspark --ctx 8192 --nothink --temp 0 \
-p "Write a Python quicksort function with comments." --tokens 100
```

Happy to run additional experiments (longer prompts, different domains, longer contexts) if useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.