antirez / antirez/ds4

CUDA --ssd-streaming: MMQ prefill tier OOMs on a 12 GB card (main b030961)

Offen
#732 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C
Sterne
22.3k
Forks
2.1k
Ø Merge
1 T. 3 Std.
Gemergte PRs (30 T.)
4

Beschreibung

On a 12 GB card the MMQ prefill tier resolves whole routed-expert tensors into VRAM and dies.
Reproduced on a clean worktree of origin/main at b030961, nothing local in it.

```./ds4 -m ./ds4flash.gguf --cuda --ssd-streaming --ctx 8192 --nothink --temp 0 -n 32 -p "..." ```

ds4: CUDA model range alloc failed for moe gate mmq (528.00 MiB): out of memory
ds4: CUDA routed_moe x quantize launch failed: out of memory
ds4: gpu layer 20 ffn batch encode failed
ds4: CUDA synchronize failed: out of memory exit 1.

Depending on which layer runs out first it is sometimes moe down mmq (1792.00 MiB) instead.
Two things visible before it dies: "CUDA loading model tensors 32.06 GiB cached" on a 12 GiB card,
and the q8->f16 pool collapsing to 0.19 GiB against its 4.00 GiB reserve.

DS4_CUDA_MMQ=0 works around it and only turns off a prefill optimisation, so streaming is usable again.

RTX 3500 Ada 12 GB (sm_89), CUDA 13.0, WSL2 Ubuntu, 31 GB RAM visible, DeepSeek-V4-Flash IQ2_XXS 80.8 GB (q2-imatrix).
NOT TESTED: larger cards, native Linux, or resident mode (the model does not fit resident here).

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.