antirez / antirez/ds4

GLM not loading, ds4: glm routed moe: unsupported types 12/12/12

Abierto
#682 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.3k
Forks
2.1k
Merge medio
1 d 3 h
PR fusionados (30 d)
4

Descripción

I used this script to download GLM :
./download_model.sh glm-antirez-q4

I try to start ds4-server with ssd-streaming :

./ds4-server --ssd-streaming
ds4: Linux cuda backend set oom_score_adj=1000
ds4: GLM memory guard refused ctx=32768 compact_cap=32768 before GLM graph allocation
ds4: streamed active model map: 5.35 GiB (full GGUF 404.35 GiB)
ds4: graph cache/scratch: 10.01 GiB (full KV 0.00 GiB, compact DSA 5.81 GiB, scratch 4.19 GiB)
ds4: required model+graph: 15.36 GiB; guard budget: 0.00 GiB (base 15.56 GiB, fraction 0.99, reserve 32.00 GiB, transient 0.00 GiB)
**ds4: set DS4_GLM_MEMORY_GUARD=0 to bypass, use a smaller --ctx, or use SSD streaming**
(started with ssd streaming parameter)

I force the start :
DS4_GLM_MEMORY_GUARD=0 ./ds4-server --ssd-streaming
ds4: Linux cuda backend set oom_score_adj=1000
ds4: CUDA backend initialized on Quadro RTX 5000 (sm_75) dev=0
ds4: SSD streaming initial cuda model map restricted to token embedding (1 spans, 0.94 GiB tensor span)
ds4: CUDA host registration skipped: out of memory
ds4: CUDA preparing model tensor mappings: 0.00 GiBds4: CUDA loading model tensors into device cache: 0.00 GiB
ds4: CUDA startup model preparation covered 0.94 GiB of tensor spans in 0.375s
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 5.81 GiB (raw 0.00 + compressed 5.81) + buffers 4.19 GiB + resident model 0.94 GiB = 10.95 GiB planned
ds4: memory detail: ctx=32768 prefill_cap=4096 raw_kv_rows=0 compressed_kv_rows=32768 backend=cuda
0804 16:44:48 ds4-server: context buffers 10245.44 MiB (ctx=32768, backend=cuda, prefill_chunk=4096, raw_kv_rows=0, compressed_kv_rows=32768)
ds4: GLM session ctx=32768 (model max=1048576); full-attention prefill/work cap=8192; compact indexed decode is used beyond the cap
ds4: GLM graph using compact DSA KV only; expanded full-attention KV cache is skipped
ds4: GLM graph allocating compact DSA cache: rows=32768 logical_ctx=32768 kv_layers=78 indexer_layers=21 f32 5.81 GiB
ds4: GLM compact indexed prefill chunk=4096 score_rows=2048 score_scratch=256.00 MiB
0804 16:44:48 ds4-server: listening on http://127.0.0.1:8000

I request some chat via API :
0804 16:45:26 ds4-server: chat ctx=0..28:28 prompt start
**ds4: CUDA loading model tensors into device cache: 2.02 GiBds4: glm routed moe: unsupported types 12/12/12**

git rev-parse --short HEAD
ae504c9

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.