antirez / antirez/ds4

DSpark net-negative on M3 Ultra + MXFP4 despite 82.8% accept rate — independent confirmation of #695 replay/verify overhead

Aperta
#733 9 commenti 1 reazione 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
22.3k
Fork
2.1k
Merge medio
1g 3h
PR unite (30g)
4

Descrizione

## Summary

Adding a cross-configuration data point for #695: DSpark is net-negative on **M3 Ultra 256GB + MXFP4** (the recommended quantization), even though the draft acceptance rate is **82.8%**. This confirms the core finding of #695 from a different hardware generation and quantization format: the bottleneck is replay + verify overhead, not draft quality.

## Environment

| Item | Value |
|------|-------|
| Chip | Apple M3 Ultra (256 GB unified memory, 819 GB/s) |
| Engine | ds4 commit `b030961` |
| Model GGUF | `DeepSeek-V4-Flash-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-mxfp4-0731.gguf` (145 GB) |
| DSpark support | `DeepSeek-V4-Flash-DSpark-support-0731.gguf` (5.6 GB) |
| Quantization | MXFP4 experts (native, bit-for-bit) + F16/Q8 non-expert |
| Context | 8192, `--nothink --temp 0` |
| Prompt | `"Write a Python quicksort function with comments."` (100 tokens) |

## Controlled Experiment (4 modes, same prompt)

| Mode | Generation (t/s) | Decode vs baseline |
|------|-----------------|-------------------|
| Baseline (no DSpark) | 39.78 | — |
| `--dspark-strict` (loaded, target-only) | 39.17 | -1.5% (loading overhead ≈ 0) |
| `--dspark` (confidence 0.7, default) | 25.47 | **-36%** |
| `--dspark --dspark-confidence 0` (forced 5-token blocks) | 21.58 | **-46%** |

The strict experiment isolates the overhead: loading the DSpark support model costs ~1.5% (within noise). The regression is entirely in the runtime decode path.

## DSpark Telemetry (`DS4_DSPARK_STATS=1`)

### Default confidence (0.7)

```
cycles=46 proposed=58 accepted_draft=48 accept_rate=82.76% avg_accept=1.043
full=12 partial=4 miss_first=2 no_draft=28 scheduler_skips=23
time_ms propose=379.8 verify=1236.4 replay=1206.8 spec_total=2460.6
target=1571.3 saved=1631.7 net_saved=-1208.7
draft_len_hist=1:2,2:4,3:5,4:2,5:5
accepted_len_hist=0:30,1:3,2:3,3:5,4:1,5:4
```

### Forced 5-token blocks (confidence 0)

```
cycles=24 proposed=120 accepted_draft=70 accept_rate=58.33% avg_accept=2.917
full=7 partial=16 miss_first=1 no_draft=0 scheduler_skips=0
time_ms propose=221.2 verify=1928.3 replay=1746.3 spec_total=3698.1
target=760.6 saved=1794.5 net_saved=-2124.8
draft_len_hist=5:24
accepted_len_hist=0:1,1:5,2:6,3:2,4:3,5:7
```

## Analysis: Where the Loss Comes From

All three time components in this run are in the same order of magnitude:

| Component | Time (ms) | Per-cycle (ms) | Notes |
|-----------|----------|----------------|-------|
| propose | 379.8 | 8.3 | Draft generation (cheap) |
| verify | 1236.4 | 26.9 | Target model verification |
| replay | 1206.8 | 26.2 | Partial-accept replay (`7fb2830`) |

**replay (1207 ms) ≈ verify (1236 ms)**. The replay cost — introduced by commit `7fb2830` ("Replay partial DSpark accepts through ordinary decode") for correctness — is as expensive as the verify pass itself. Combined, replay + verify = 2443 ms, while accepted tokens only save 1632 ms.

This is consistent with the finding in #695: the scheduler's break-even model may not account for replay cost, preventing convergence to baseline even when draft quality is good.

## What This Confirms (and What It Doesn't)

**Confirms**: Draft quality is not the bottleneck on MXFP4. An 82.8% acceptance rate — higher than expected — still yields a net loss. The regression is driven by replay + verify overhead.

**Does not claim**: This report does not compare absolute timing against #695, as the test scales differ (#695 covers a multi-thousand-token corpus; this is a 100-token single prompt). Per-cycle costs are not directly comparable across different draft block sizes and corpus lengths. The shared conclusion is qualitative: both reports independently observe net-negative results despite healthy acceptance rates.

## Scheduler Behavior

In default mode (confidence 0.7), the scheduler skips 23 of 46 cycles (`no_draft=28`). This is the correct behavior — when forced to propose every cycle (confidence 0), net loss worsens from -1209 ms to -2125 ms. The scheduler is already doing damage control, but as #695 notes, the break-even model may not fully account for replay cost, preventing convergence to baseline.

## Repro

```bash
# Baseline
./ds4 -m --ctx 8192 --nothink --temp 0 \
-p "Write a Python quicksort function with comments." --tokens 100

# DSpark with telemetry
DS4_DSPARK_STATS=1 ./ds4 \
-m \
--mtp gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \
--dspark --ctx 8192 --nothink --temp 0 \
-p "Write a Python quicksort function with comments." --tokens 100
```

Happy to run additional experiments (longer prompts, different domains, longer contexts) if useful.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.