GLM not loading, ds4: glm routed moe: unsupported types 12/12/12
- Langage dominant
- C
- Étoiles
- 22.3k
- Forks
- 2.1k
- Merge moyen
- 1 j 3 h
- PR mergées (30 j)
- 4
Description
I used this script to download GLM :
./download_model.sh glm-antirez-q4
I try to start ds4-server with ssd-streaming :
./ds4-server --ssd-streaming
ds4: Linux cuda backend set oom_score_adj=1000
ds4: GLM memory guard refused ctx=32768 compact_cap=32768 before GLM graph allocation
ds4: streamed active model map: 5.35 GiB (full GGUF 404.35 GiB)
ds4: graph cache/scratch: 10.01 GiB (full KV 0.00 GiB, compact DSA 5.81 GiB, scratch 4.19 GiB)
ds4: required model+graph: 15.36 GiB; guard budget: 0.00 GiB (base 15.56 GiB, fraction 0.99, reserve 32.00 GiB, transient 0.00 GiB)
**ds4: set DS4_GLM_MEMORY_GUARD=0 to bypass, use a smaller --ctx, or use SSD streaming**
(started with ssd streaming parameter)
I force the start :
DS4_GLM_MEMORY_GUARD=0 ./ds4-server --ssd-streaming
ds4: Linux cuda backend set oom_score_adj=1000
ds4: CUDA backend initialized on Quadro RTX 5000 (sm_75) dev=0
ds4: SSD streaming initial cuda model map restricted to token embedding (1 spans, 0.94 GiB tensor span)
ds4: CUDA host registration skipped: out of memory
ds4: CUDA preparing model tensor mappings: 0.00 GiBds4: CUDA loading model tensors into device cache: 0.00 GiB
ds4: CUDA startup model preparation covered 0.94 GiB of tensor spans in 0.375s
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 5.81 GiB (raw 0.00 + compressed 5.81) + buffers 4.19 GiB + resident model 0.94 GiB = 10.95 GiB planned
ds4: memory detail: ctx=32768 prefill_cap=4096 raw_kv_rows=0 compressed_kv_rows=32768 backend=cuda
0804 16:44:48 ds4-server: context buffers 10245.44 MiB (ctx=32768, backend=cuda, prefill_chunk=4096, raw_kv_rows=0, compressed_kv_rows=32768)
ds4: GLM session ctx=32768 (model max=1048576); full-attention prefill/work cap=8192; compact indexed decode is used beyond the cap
ds4: GLM graph using compact DSA KV only; expanded full-attention KV cache is skipped
ds4: GLM graph allocating compact DSA cache: rows=32768 logical_ctx=32768 kv_layers=78 indexer_layers=21 f32 5.81 GiB
ds4: GLM compact indexed prefill chunk=4096 score_rows=2048 score_scratch=256.00 MiB
0804 16:44:48 ds4-server: listening on http://127.0.0.1:8000
I request some chat via API :
0804 16:45:26 ds4-server: chat ctx=0..28:28 prompt start
**ds4: CUDA loading model tensors into device cache: 2.02 GiBds4: glm routed moe: unsupported types 12/12/12**
git rev-parse --short HEAD
ae504c9
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.