antirez / antirez/ds4

GLM-5.3 Flash Q4_K + CUDA + --ssd-streaming fails: routed MoE unsupported types 12/12/12

Open
#907 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

Environment: NVIDIA H100 PCIe 80 GB (sm_90), ~235 GiB host RAM, NVIDIA driver 580.173.02, CUDA 13.0, DS4 branch glm-5.3-flash, built with make cuda CUDA_ARCH=sm_90 -j$(nproc).

Model: GLM-5.3-Flash-Q4_K.gguf, downloaded with ./download_model.sh glm53-q4.

Repro:

./ds4 \
-m /models/ds4-models/GLM-5.3-Flash-Q4_K.gguf \
--cuda \
--ssd-streaming \
-c 32768 \
-n 32 \
--temp 0 \
-p "Reply only with OK."

Failure:

ds4: CUDA backend initialized on NVIDIA H100 PCIe (sm_90) dev=0
ds4: SSD streaming initial cuda model map restricted to token embedding
...
processing 23 input tokens: 1/23 (4.3%)
ds4: glm routed moe: unsupported types 12/12/12
ds4: GLM prefill failed

Without --ssd-streaming, model parsing works but the GLM memory guard correctly refuses full residency because the model map is ~177.77 GiB on an 80 GB GPU.

This looks like a CUDA routed-MoE dispatch/type-support gap for the GLM-5.3 Q4_K layout rather than a loading or memory-planning issue.

Happy to test patches or provide more diagnostics on the H100.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.