antirez / antirez/ds4

CUDA --ssd-streaming: MMQ prefill tier OOMs on a 12 GB card (main b030961)

Ouverte
#732 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
C
Étoiles
22.3k
Forks
2.1k
Merge moyen
1 j 3 h
PR mergées (30 j)
4

Description

On a 12 GB card the MMQ prefill tier resolves whole routed-expert tensors into VRAM and dies.
Reproduced on a clean worktree of origin/main at b030961, nothing local in it.

```./ds4 -m ./ds4flash.gguf --cuda --ssd-streaming --ctx 8192 --nothink --temp 0 -n 32 -p "..." ```

ds4: CUDA model range alloc failed for moe gate mmq (528.00 MiB): out of memory
ds4: CUDA routed_moe x quantize launch failed: out of memory
ds4: gpu layer 20 ffn batch encode failed
ds4: CUDA synchronize failed: out of memory exit 1.

Depending on which layer runs out first it is sometimes moe down mmq (1792.00 MiB) instead.
Two things visible before it dies: "CUDA loading model tensors 32.06 GiB cached" on a 12 GiB card,
and the q8->f16 pool collapsing to 0.19 GiB against its 4.00 GiB reserve.

DS4_CUDA_MMQ=0 works around it and only turns off a prefill optimisation, so streaming is usable again.

RTX 3500 Ada 12 GB (sm_89), CUDA 13.0, WSL2 Ubuntu, 31 GB RAM visible, DeepSeek-V4-Flash IQ2_XXS 80.8 GB (q2-imatrix).
NOT TESTED: larger cards, native Linux, or resident mode (the model does not fit resident here).

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.