antirez / antirez/ds4

DSpark never produces a draft on ROCm / gfx1151: 1197 cycles, proposed=0, prop_hidden=0.000

Abierto
#597 3 comentarios 1 reacción 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.3k
Forks
2.1k
Merge medio
1 d 3 h
PR fusionados (30 d)
4

Descripción

## Summary

With `--dspark` enabled on a Strix Halo (gfx1151) ROCm build, the DSpark
speculative path is entered on every decode cycle but never emits a single
proposed token. `DS4_DSPARK_STATS=1` shows the proposal pipeline running
`prop_setup` and `prop_chain`, then stopping: every stage from `prop_hidden`
onward stays at exactly `0.000` ms. `draft_len_hist=none`.

Net effect is a small loss, never a gain. The adaptive scheduler correctly
suppresses most attempts, which is why the cost is normally invisible.

## Environment

| | |
|---|---|
| CPU / GPU | AMD Ryzen AI Max+ 395, Radeon 8060S Graphics (`gfx1151`), 128 GB unified |
| Host OS | Arch Linux |
| Container | `archlinux:latest`, Docker |
| ROCm | 7.14.0, TheRock tarball `therock-dist-linux-gfx1151-7.14.0.tar.gz` extracted to `/opt/rocm` |
| ds4 commit | `0a7ad776b9068348e6cb09df8cafa9cadd285298` |
| Build | `make strix-halo` |
| Model | `DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf` |
| Support model | `DeepSeek-V4-Flash-DSpark-support.gguf` |

Startup reports the support model as healthy and capture as active:

```
ds4: DSpark support model detected: /models/DeepSeek-V4-Flash-DSpark-support.gguf
(stages=3 block=5 markov_rank=256 tensors=81 missing=0 invalid=0 metadata_errors=0)
ds4: ROCm backend initialized on AMD Radeon 8060S Graphics (sm_115)
ds4: DSpark target-hidden capture enabled: layers=40,41,42
```

## Reproduction

Server:

```
ds4-server \
-m DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
--mtp DeepSeek-V4-Flash-DSpark-support.gguf \
--dspark \
--host 0.0.0.0 --ctx 200000
```

with `DS4_DSPARK_STATS=1` in the environment.

Three identical greedy requests, minimal payload, no client-side sampler
parameters:

```bash
for i in 1 2 3; do
curl -s http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"deepseek-v4-flash",
"messages":[{"role":"user","content":"Write a Python LinkedList class with push, pop, reverse, __len__ and __iter__. Include docstrings and type hints."}],
"temperature":0,
"max_tokens":400}' >/dev/null
done
```

Then stop the server so the session is destroyed and stats are flushed.

## Observed

### Run A — default scheduler

```
DSpark stats cycles=360 first_tokens=360 proposed=0 accepted_draft=0
accept_rate=0.00% avg_accept=0.000 full=0 partial=0 miss_first=0 no_draft=360
no_room=0 invalid=0 scheduler_skips=275 tail_skips=8 verifier_unavailable=0
errors=0 time_ms propose=162.318 prop_stage0=0.002 prop_setup=25.770
prop_cache=2.956 prop_chain=133.404 prop_hidden=0.000 prop_conf0=0.000
prop_logits=0.000 prop_markov=0.000 prop_confidence=0.000 snapshot=0.000
verify=0.000 target=85230.997 saved=0.000 net_saved=-162.420
draft_len_hist=none accepted_len_hist=0:360
```

### Run B — `DS4_DSPARK_SCHEDULER=0`

```
DSpark stats cycles=1197 first_tokens=1197 proposed=0 accepted_draft=0
accept_rate=0.00% avg_accept=0.000 full=0 partial=0 miss_first=0 no_draft=1197
no_room=0 invalid=0 scheduler_skips=0 tail_skips=0 verifier_unavailable=0
errors=0 time_ms propose=2207.187 prop_stage0=0.028 prop_setup=360.438
prop_cache=12.962 prop_chain=1832.794 prop_hidden=0.000 prop_conf0=0.000
prop_logits=0.000 prop_markov=0.000 prop_confidence=0.000 snapshot=0.000
verify=0.000 spec_total=0.100 target=73028.190 saved=0.000 net_saved=-2207.287
draft_len_hist=none accepted_len_hist=0:1197
```

With the scheduler disabled every cycle reaches the proposal path
(`scheduler_skips=0`, `tail_skips=0`) and every cycle still ends in
`no_draft`.

Decode throughput, unchanged across every configuration tried:

| configuration | decode |
|---|---|
| `--dspark`, default scheduler | 16.2–16.3 t/s |
| `--dspark`, `DS4_DSPARK_SCHEDULER=0` | 15.85–16.10 t/s |
| no `--dspark` | 16.2–16.3 t/s |

The ~3% drop in run B is the forced proposal overhead: 2207 ms against
73028 ms of target decode.

## Where it stops

`prop_setup` and `prop_chain` accumulate real time (360 ms and 1833 ms in
run B). Everything downstream is exactly zero:

```
prop_hidden=0.000 prop_conf0=0.000 prop_logits=0.000
prop_markov=0.000 prop_confidence=0.000 snapshot=0.000 verify=0.000
```

So the chain stage runs to completion and the hidden-state consumption never
starts, even though capture is reported as enabled for layers 40, 41, 42.
`errors=0`, `invalid=0` and `verifier_unavailable=0` — nothing is failing
loudly, the draft simply never forms.

## Ruled out

- **Sampled decoding.** Requests use `temperature: 0` with no other sampler
parameters, issued by curl directly against the server.
- **Scheduler suppression.** `DS4_DSPARK_SCHEDULER=0`, `scheduler_skips=0`.
- **Confidence threshold.** `proposed=0`, so no suffix is ever pruned; the
threshold is never reached.
- **Support model integrity.** `missing=0 invalid=0 metadata_errors=0`.
- **Capacity.** `no_room=0`.

## Not ruled out

The build carries a community patch that changes the ROCm decode kernels
(routing `n_tok == 1` matmuls through the prequantized dp4a path, and
disabling the batch hc-norm fusion which otherwise produces corrupted output
on this GPU). It does not touch `strict_dspark` or the proposal pipeline, but
it does modify `rocm/ds4_rocm_attention_launch.cuh`, so it cannot be excluded
as a contributor without a clean-tree comparison. Happy to rerun on an
unpatched build with `DS4_DSPARK_STATS=1` if that is useful — note that an
unpatched build needs `DS4_METAL_DISABLE_BATCH_HC_NORM_FUSION=1` to produce
coherent output at all on gfx1151.

## Expected

Some non-zero `proposed` count, and `prop_hidden` accumulating time.

## Impact

Low severity on its own: DSpark is documented as experimental and opt-in, the
scheduler keeps the wasted work near 0.2% in normal operation, and correctness
is unaffected since the target model stays authoritative.

One caveat for anyone benchmarking this. Registering the support model map
sets `g_q8_f16_disabled_for_multi_model`, which disables the expanded Q8→F16
cache — so toggling `--dspark` silently toggles that cache as well. Measured
GTT on this machine, at rest after startup:

| configuration | GTT used | decode |
|---|---|---|
| `--mtp` + `--dspark` | 99.98 GiB | 16.31 t/s |
| neither | 106.96 GiB | 16.38 t/s |

Enabling a feature that adds a 5.58 GiB resident model *reduces* total GTT by
7 GiB, because the cache it suppresses is worth about 12.6 GiB here. An A/B on
`--dspark` alone therefore moves two variables at once; setting
`DS4_CUDA_NO_Q8_F16_CACHE=1` on both sides isolates the speculation path.

Secondary observation, unrelated to the bug itself: on this UMA system that
cache costs 12.6 GiB for ~0.4% of decode throughput, which is inside
run-to-run noise. Worth revisiting as a default on unified-memory parts, where
it competes directly with KV and context tensors instead of living in separate
VRAM.

Happy to provide `DS4_DSPARK_PROBE` output, run with
`--dspark-confidence 0`, or test any patch.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.