antirez / antirez/ds4

GLM-5.3 Flash Q4_K + CUDA + --ssd-streaming fails: routed MoE unsupported types 12/12/12

未关闭
#907 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C
星标
22.3k
派生
2.1k
平均合并
1 天 3 小时
30 天内合并 PR
4

描述

Environment: NVIDIA H100 PCIe 80 GB (sm_90), ~235 GiB host RAM, NVIDIA driver 580.173.02, CUDA 13.0, DS4 branch glm-5.3-flash, built with make cuda CUDA_ARCH=sm_90 -j$(nproc).

Model: GLM-5.3-Flash-Q4_K.gguf, downloaded with ./download_model.sh glm53-q4.

Repro:

./ds4 \
-m /models/ds4-models/GLM-5.3-Flash-Q4_K.gguf \
--cuda \
--ssd-streaming \
-c 32768 \
-n 32 \
--temp 0 \
-p "Reply only with OK."

Failure:

ds4: CUDA backend initialized on NVIDIA H100 PCIe (sm_90) dev=0
ds4: SSD streaming initial cuda model map restricted to token embedding
...
processing 23 input tokens: 1/23 (4.3%)
ds4: glm routed moe: unsupported types 12/12/12
ds4: GLM prefill failed

Without --ssd-streaming, model parsing works but the GLM memory guard correctly refuses full residency because the model map is ~177.77 GiB on an 80 GB GPU.

This looks like a CUDA routed-MoE dispatch/type-support gap for the GLM-5.3 Q4_K layout rather than a loading or memory-planning issue.

Happy to test patches or provide more diagnostics on the H100.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。