antirez / antirez/ds4

CUDA --ssd-streaming: illegal memory access at first 2048-token prefill in ds4-bench (RTX 3090, layer-major FFN encode)

Ouverte
#881 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
C
Étoiles
22.3k
Forks
2.1k
Merge moyen
1 j 3 h
PR mergées (30 j)
4

Description

## Summary

`ds4-bench` with CUDA `--ssd-streaming` crashes with an **illegal memory access at the very first prefill** (2048 tokens), on a plain RTX 3090. The failing stage is the layer-major streaming prefill, FFN encode of layer 2. Reproduced 3/3 runs; while a `ds4-server` on the same build, same box, same model serves short-prompt chat completions for hours without issue (we ran one idle for 4 h during the #836 measurement).

Likely the same family as the mid-prefill poisoning on GB10 (#759 / #847), but this signature is different: consumer sm_86, first prefill of a fresh process, deterministic, and reachable with the project's own bench tool — so it may be the easiest handle on this class of bug so far.

## Environment (per CONTRIBUTING.md)

- RunPod pod, 1× RTX 3090 24 GB (sm_86), driver 580.65.06, 251 GB RAM
- `make cuda CUDA_ARCH=sm_86`, CUDA 12.4 toolkit, commit `c1d4597`
- Model quant: `ds4f-q2` (DeepSeek-V4-Flash-IQ2XXS-...-0731, 81 GB)

## Repro (deterministic, 3/3)

```sh
./ds4-bench -m ./ds4flash.gguf --cuda --ssd-streaming \
--prompt-file speed-bench/promessi_sposi.txt \
--ctx-start 2048 --ctx-max 8192 --step-incr 2048 \
--gen-tokens 64 --csv /tmp/speed.csv
```

Output:

```
ds4: CUDA model range alloc failed for hc_scale (0.00 MiB): an illegal memory access was encountered
ds4: CUDA model range copy failed for hc_base at 0.00 MiB: an illegal memory access was encountered
ds4: CUDA model range alloc failed for hc_base (0.00 MiB): an illegal memory access was encountered
ds4: gpu layer 2 ffn batch encode failed
ds4: gpu layer-major prefill layer 2 encode failed
ds4: CUDA synchronize failed: an illegal memory access was encountered
ds4-bench: prefill to 2048 failed: cuda prefill failed
```

(The `model range alloc failed` lines are the poisoned-context cascade; the first IMA happens earlier in the layer-major prefill.)

## Notes

- Identical behavior on unpatched `c1d4597` and with PR #872 applied — that patch is host-side shape validation in the attention prefill fallback and is unrelated; mentioning it only because we found this crash while trying to run the before/after bench for that PR.
- Without `--ssd-streaming` the model cannot fit a 24 GB card (arena alloc OOM), so on this class of hardware the bench currently has no working path.
- Happy to run diagnostics on a rented pod if a compute-sanitizer trace or an env-var A/B (`DS4_CUDA_NO_TOPK_STREAM`, etc.) would help — CUDA 13.0 compute-sanitizer works on these drivers, the 12.4 one does not start.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.