CUDA --ssd-streaming: MMQ prefill tier OOMs on a 12 GB card (main b030961)
- Lenguaje dominante
- C
- Estrellas
- 22.3k
- Forks
- 2.1k
- Merge medio
- 1 d 3 h
- PR fusionados (30 d)
- 4
Descripción
On a 12 GB card the MMQ prefill tier resolves whole routed-expert tensors into VRAM and dies.
Reproduced on a clean worktree of origin/main at b030961, nothing local in it.
```./ds4 -m ./ds4flash.gguf --cuda --ssd-streaming --ctx 8192 --nothink --temp 0 -n 32 -p "..." ```
ds4: CUDA model range alloc failed for moe gate mmq (528.00 MiB): out of memory
ds4: CUDA routed_moe x quantize launch failed: out of memory
ds4: gpu layer 20 ffn batch encode failed
ds4: CUDA synchronize failed: out of memory exit 1.
Depending on which layer runs out first it is sometimes moe down mmq (1792.00 MiB) instead.
Two things visible before it dies: "CUDA loading model tensors 32.06 GiB cached" on a 12 GiB card,
and the q8->f16 pool collapsing to 0.19 GiB against its 4.00 GiB reserve.
DS4_CUDA_MMQ=0 works around it and only turns off a prefill optimisation, so streaming is usable again.
RTX 3500 Ada 12 GB (sm_89), CUDA 13.0, WSL2 Ubuntu, 31 GB RAM visible, DeepSeek-V4-Flash IQ2_XXS 80.8 GB (q2-imatrix).
NOT TESTED: larger cards, native Linux, or resident mode (the model does not fit resident here).
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.