GLM-5.3 Flash Q4_K + CUDA + --ssd-streaming fails: routed MoE unsupported types 12/12/12
- Lenguaje dominante
- C
- Estrellas
- 22.3k
- Forks
- 2.1k
- Merge medio
- 1 d 3 h
- PR fusionados (30 d)
- 4
Descripción
Environment: NVIDIA H100 PCIe 80 GB (sm_90), ~235 GiB host RAM, NVIDIA driver 580.173.02, CUDA 13.0, DS4 branch glm-5.3-flash, built with make cuda CUDA_ARCH=sm_90 -j$(nproc).
Model: GLM-5.3-Flash-Q4_K.gguf, downloaded with ./download_model.sh glm53-q4.
Repro:
./ds4 \
-m /models/ds4-models/GLM-5.3-Flash-Q4_K.gguf \
--cuda \
--ssd-streaming \
-c 32768 \
-n 32 \
--temp 0 \
-p "Reply only with OK."
Failure:
ds4: CUDA backend initialized on NVIDIA H100 PCIe (sm_90) dev=0
ds4: SSD streaming initial cuda model map restricted to token embedding
...
processing 23 input tokens: 1/23 (4.3%)
ds4: glm routed moe: unsupported types 12/12/12
ds4: GLM prefill failed
Without --ssd-streaming, model parsing works but the GLM memory guard correctly refuses full residency because the model map is ~177.77 GiB on an 80 GB GPU.
This looks like a CUDA routed-MoE dispatch/type-support gap for the GLM-5.3 Q4_K layout rather than a loading or memory-planning issue.
Happy to test patches or provide more diagnostics on the H100.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.