antirez / antirez/ds4

CUDA --ssd-streaming: illegal memory access at first 2048-token prefill in ds4-bench (RTX 3090, layer-major FFN encode)

Offen
#881 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C
Sterne
22.3k
Forks
2.1k
Ø Merge
1 T. 3 Std.
Gemergte PRs (30 T.)
4

Beschreibung

## Summary

`ds4-bench` with CUDA `--ssd-streaming` crashes with an **illegal memory access at the very first prefill** (2048 tokens), on a plain RTX 3090. The failing stage is the layer-major streaming prefill, FFN encode of layer 2. Reproduced 3/3 runs; while a `ds4-server` on the same build, same box, same model serves short-prompt chat completions for hours without issue (we ran one idle for 4 h during the #836 measurement).

Likely the same family as the mid-prefill poisoning on GB10 (#759 / #847), but this signature is different: consumer sm_86, first prefill of a fresh process, deterministic, and reachable with the project's own bench tool — so it may be the easiest handle on this class of bug so far.

## Environment (per CONTRIBUTING.md)

- RunPod pod, 1× RTX 3090 24 GB (sm_86), driver 580.65.06, 251 GB RAM
- `make cuda CUDA_ARCH=sm_86`, CUDA 12.4 toolkit, commit `c1d4597`
- Model quant: `ds4f-q2` (DeepSeek-V4-Flash-IQ2XXS-...-0731, 81 GB)

## Repro (deterministic, 3/3)

```sh
./ds4-bench -m ./ds4flash.gguf --cuda --ssd-streaming \
--prompt-file speed-bench/promessi_sposi.txt \
--ctx-start 2048 --ctx-max 8192 --step-incr 2048 \
--gen-tokens 64 --csv /tmp/speed.csv
```

Output:

```
ds4: CUDA model range alloc failed for hc_scale (0.00 MiB): an illegal memory access was encountered
ds4: CUDA model range copy failed for hc_base at 0.00 MiB: an illegal memory access was encountered
ds4: CUDA model range alloc failed for hc_base (0.00 MiB): an illegal memory access was encountered
ds4: gpu layer 2 ffn batch encode failed
ds4: gpu layer-major prefill layer 2 encode failed
ds4: CUDA synchronize failed: an illegal memory access was encountered
ds4-bench: prefill to 2048 failed: cuda prefill failed
```

(The `model range alloc failed` lines are the poisoned-context cascade; the first IMA happens earlier in the layer-major prefill.)

## Notes

- Identical behavior on unpatched `c1d4597` and with PR #872 applied — that patch is host-side shape validation in the attention prefill fallback and is unrelated; mentioning it only because we found this crash while trying to run the before/after bench for that PR.
- Without `--ssd-streaming` the model cannot fit a 24 GB card (arena alloc OOM), so on this class of hardware the bench currently has no working path.
- Happy to run diagnostics on a rented pod if a compute-sanitizer trace or an env-var A/B (`DS4_CUDA_NO_TOPK_STREAM`, etc.) would help — CUDA 13.0 compute-sanitizer works on these drivers, the 12.4 one does not start.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.