antirez / antirez/ds4

DSpark never produces a draft on ROCm / gfx1151: 1197 cycles, proposed=0, prop_hidden=0.000

Open
#597 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

## Summary

With `--dspark` enabled on a Strix Halo (gfx1151) ROCm build, the DSpark
speculative path is entered on every decode cycle but never emits a single
proposed token. `DS4_DSPARK_STATS=1` shows the proposal pipeline running
`prop_setup` and `prop_chain`, then stopping: every stage from `prop_hidden`
onward stays at exactly `0.000` ms. `draft_len_hist=none`.

Net effect is a small loss, never a gain. The adaptive scheduler correctly
suppresses most attempts, which is why the cost is normally invisible.

## Environment

| | |
|---|---|
| CPU / GPU | AMD Ryzen AI Max+ 395, Radeon 8060S Graphics (`gfx1151`), 128 GB unified |
| Host OS | Arch Linux |
| Container | `archlinux:latest`, Docker |
| ROCm | 7.14.0, TheRock tarball `therock-dist-linux-gfx1151-7.14.0.tar.gz` extracted to `/opt/rocm` |
| ds4 commit | `0a7ad776b9068348e6cb09df8cafa9cadd285298` |
| Build | `make strix-halo` |
| Model | `DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf` |
| Support model | `DeepSeek-V4-Flash-DSpark-support.gguf` |

Startup reports the support model as healthy and capture as active:

```
ds4: DSpark support model detected: /models/DeepSeek-V4-Flash-DSpark-support.gguf
(stages=3 block=5 markov_rank=256 tensors=81 missing=0 invalid=0 metadata_errors=0)
ds4: ROCm backend initialized on AMD Radeon 8060S Graphics (sm_115)
ds4: DSpark target-hidden capture enabled: layers=40,41,42
```

## Reproduction

Server:

```
ds4-server \
-m DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
--mtp DeepSeek-V4-Flash-DSpark-support.gguf \
--dspark \
--host 0.0.0.0 --ctx 200000
```

with `DS4_DSPARK_STATS=1` in the environment.

Three identical greedy requests, minimal payload, no client-side sampler
parameters:

```bash
for i in 1 2 3; do
curl -s http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"deepseek-v4-flash",
"messages":[{"role":"user","content":"Write a Python LinkedList class with push, pop, reverse, __len__ and __iter__. Include docstrings and type hints."}],
"temperature":0,
"max_tokens":400}' >/dev/null
done
```

Then stop the server so the session is destroyed and stats are flushed.

## Observed

### Run A — default scheduler

```
DSpark stats cycles=360 first_tokens=360 proposed=0 accepted_draft=0
accept_rate=0.00% avg_accept=0.000 full=0 partial=0 miss_first=0 no_draft=360
no_room=0 invalid=0 scheduler_skips=275 tail_skips=8 verifier_unavailable=0
errors=0 time_ms propose=162.318 prop_stage0=0.002 prop_setup=25.770
prop_cache=2.956 prop_chain=133.404 prop_hidden=0.000 prop_conf0=0.000
prop_logits=0.000 prop_markov=0.000 prop_confidence=0.000 snapshot=0.000
verify=0.000 target=85230.997 saved=0.000 net_saved=-162.420
draft_len_hist=none accepted_len_hist=0:360
```

### Run B — `DS4_DSPARK_SCHEDULER=0`

```
DSpark stats cycles=1197 first_tokens=1197 proposed=0 accepted_draft=0
accept_rate=0.00% avg_accept=0.000 full=0 partial=0 miss_first=0 no_draft=1197
no_room=0 invalid=0 scheduler_skips=0 tail_skips=0 verifier_unavailable=0
errors=0 time_ms propose=2207.187 prop_stage0=0.028 prop_setup=360.438
prop_cache=12.962 prop_chain=1832.794 prop_hidden=0.000 prop_conf0=0.000
prop_logits=0.000 prop_markov=0.000 prop_confidence=0.000 snapshot=0.000
verify=0.000 spec_total=0.100 target=73028.190 saved=0.000 net_saved=-2207.287
draft_len_hist=none accepted_len_hist=0:1197
```

With the scheduler disabled every cycle reaches the proposal path
(`scheduler_skips=0`, `tail_skips=0`) and every cycle still ends in
`no_draft`.

Decode throughput, unchanged across every configuration tried:

| configuration | decode |
|---|---|
| `--dspark`, default scheduler | 16.2–16.3 t/s |
| `--dspark`, `DS4_DSPARK_SCHEDULER=0` | 15.85–16.10 t/s |
| no `--dspark` | 16.2–16.3 t/s |

The ~3% drop in run B is the forced proposal overhead: 2207 ms against
73028 ms of target decode.

## Where it stops

`prop_setup` and `prop_chain` accumulate real time (360 ms and 1833 ms in
run B). Everything downstream is exactly zero:

```
prop_hidden=0.000 prop_conf0=0.000 prop_logits=0.000
prop_markov=0.000 prop_confidence=0.000 snapshot=0.000 verify=0.000
```

So the chain stage runs to completion and the hidden-state consumption never
starts, even though capture is reported as enabled for layers 40, 41, 42.
`errors=0`, `invalid=0` and `verifier_unavailable=0` — nothing is failing
loudly, the draft simply never forms.

## Ruled out

- **Sampled decoding.** Requests use `temperature: 0` with no other sampler
parameters, issued by curl directly against the server.
- **Scheduler suppression.** `DS4_DSPARK_SCHEDULER=0`, `scheduler_skips=0`.
- **Confidence threshold.** `proposed=0`, so no suffix is ever pruned; the
threshold is never reached.
- **Support model integrity.** `missing=0 invalid=0 metadata_errors=0`.
- **Capacity.** `no_room=0`.

## Not ruled out

The build carries a community patch that changes the ROCm decode kernels
(routing `n_tok == 1` matmuls through the prequantized dp4a path, and
disabling the batch hc-norm fusion which otherwise produces corrupted output
on this GPU). It does not touch `strict_dspark` or the proposal pipeline, but
it does modify `rocm/ds4_rocm_attention_launch.cuh`, so it cannot be excluded
as a contributor without a clean-tree comparison. Happy to rerun on an
unpatched build with `DS4_DSPARK_STATS=1` if that is useful — note that an
unpatched build needs `DS4_METAL_DISABLE_BATCH_HC_NORM_FUSION=1` to produce
coherent output at all on gfx1151.

## Expected

Some non-zero `proposed` count, and `prop_hidden` accumulating time.

## Impact

Low severity on its own: DSpark is documented as experimental and opt-in, the
scheduler keeps the wasted work near 0.2% in normal operation, and correctness
is unaffected since the target model stays authoritative.

One caveat for anyone benchmarking this. Registering the support model map
sets `g_q8_f16_disabled_for_multi_model`, which disables the expanded Q8→F16
cache — so toggling `--dspark` silently toggles that cache as well. Measured
GTT on this machine, at rest after startup:

| configuration | GTT used | decode |
|---|---|---|
| `--mtp` + `--dspark` | 99.98 GiB | 16.31 t/s |
| neither | 106.96 GiB | 16.38 t/s |

Enabling a feature that adds a 5.58 GiB resident model *reduces* total GTT by
7 GiB, because the cache it suppresses is worth about 12.6 GiB here. An A/B on
`--dspark` alone therefore moves two variables at once; setting
`DS4_CUDA_NO_Q8_F16_CACHE=1` on both sides isolates the speculation path.

Secondary observation, unrelated to the bug itself: on this UMA system that
cache costs 12.6 GiB for ~0.4% of decode throughput, which is inside
run-to-run noise. Worth revisiting as a default on unified-memory parts, where
it competes directly with KV and context tensors instead of living in separate
VRAM.

Happy to provide `DS4_DSPARK_PROBE` output, run with
`--dspark-confidence 0`, or test any patch.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.