CUDA --ssd-streaming: MMQ prefill tier OOMs on a 12 GB card (main b030961)
- Vorherrschende Sprache
- C
- Sterne
- 22.3k
- Forks
- 2.1k
- Ø Merge
- 1 T. 3 Std.
- Gemergte PRs (30 T.)
- 4
Beschreibung
On a 12 GB card the MMQ prefill tier resolves whole routed-expert tensors into VRAM and dies.
Reproduced on a clean worktree of origin/main at b030961, nothing local in it.
```./ds4 -m ./ds4flash.gguf --cuda --ssd-streaming --ctx 8192 --nothink --temp 0 -n 32 -p "..." ```
ds4: CUDA model range alloc failed for moe gate mmq (528.00 MiB): out of memory
ds4: CUDA routed_moe x quantize launch failed: out of memory
ds4: gpu layer 20 ffn batch encode failed
ds4: CUDA synchronize failed: out of memory exit 1.
Depending on which layer runs out first it is sometimes moe down mmq (1792.00 MiB) instead.
Two things visible before it dies: "CUDA loading model tensors 32.06 GiB cached" on a 12 GiB card,
and the q8->f16 pool collapsing to 0.19 GiB against its 4.00 GiB reserve.
DS4_CUDA_MMQ=0 works around it and only turns off a prefill optimisation, so streaming is usable again.
RTX 3500 Ada 12 GB (sm_89), CUDA 13.0, WSL2 Ubuntu, 31 GB RAM visible, DeepSeek-V4-Flash IQ2_XXS 80.8 GB (q2-imatrix).
NOT TESTED: larger cards, native Linux, or resident mode (the model does not fit resident here).
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.