antirez / antirez/ds4

GLM-5.2 Q4: prefill fails at ≥64K context on M3 Ultra 512GB (KV payload staging failed)

Abierto
#998 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.3k
Forks
2.1k
Merge medio
1 d 3 h
PR fusionados (30 d)
4

Descripción

## Environment
- **Hardware**: Apple M3 Ultra, 512GB unified memory, 819GB/s bandwidth
- **Model**: GLM-5.2-UD-Q4_K_RoutedQ4K.gguf (404GB, from antirez/GLM-5.2-GGUF on HuggingFace)
- **Engine**: ds4-server, built from latest main (Metal backend)
- **Context**: --ctx 524288 (512K)
- **Command**: ./ds4-server -m GLM-5.2-UD-Q4_K_RoutedQ4K.gguf --ctx 524288 --port 8321 --kv-disk-dir ~/.ds4/server-kv --kv-disk-space-mb 262144

## What works
- Short context (2K, 16K): prefill and decode work correctly
- 2K: prefill 116 t/s, decode 16.0 t/s
- 16K: prefill 112 t/s, decode 14.2 t/s
- Memory is sufficient (model 404GB + graph ~50GB = ~455GB, well within 512GB)
- Quality: needle tests pass when thinking mode is disabled

## What fails
At ≥64K context, prefill fails with:
```
ds4-server: kv cache skipped tokens=10240 reason=continued because KV payload staging failed: se...
ds4-server: kv cache discarded reason=prefill-failed
```

After the failure, the engine stops responding to new requests (appears hung).

Reproducible on a fresh process. Same hardware runs DeepSeek V4 Flash MXFP4 at 1M context without issues.

## Log excerpt
```
0907 02:11:17 ds4-server: kv cache skipped tokens=10240 reason=continued because KV payload staging failed: se...
0907 02:35:35 ds4-server: chat ctx=2048..79031:76983 prompt start
0907 02:35:35 ds4-server: kv cache discarded reason=prefill-failed file=...kv
```

## Notes
- This might be related to GLM's IndexShare architecture or the compact DSA KV layout at larger contexts
- The model loads fine and memory guard passes at 512K ctx (required: ~455GB, budget: 480GB)
- Happy to provide more logs or test patches

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.