GLM-5.2 (glm-dsa) prefill fails on CUDA / DGX Spark GB10 (sm_121): "cuda GLM prefill failed" with --ssd-streaming
- Langage dominant
- C
- Étoiles
- 22.3k
- Forks
- 2.1k
- Merge moyen
- 1 j 3 h
- PR mergées (30 j)
- 4
Description
## Summary
On a DGX Spark (GB10, sm_121, CUDA 13.0, built with `make cuda-spark`), GLM-5.2 IQ2_XXS single-node with `--ssd-streaming` loads and allocates the GLM compact-DSA graph, but then fails at prefill:
```
ds4: prompt processing failed: cuda GLM prefill failed
```
There is no OOM (~116 GiB free at the time). DeepSeek-V4-Flash is the CUDA path this engine primarily targets and builds fine; this looks GLM-specific on the CUDA backend.
## Environment
- Hardware: NVIDIA DGX Spark / GB10 (sm_121), 128 GiB unified memory
- Build: `make cuda-spark`, nvcc CUDA 13.0, Ubuntu 24.04, driver 580.x
- Model: `antirez/GLM-5.2-GGUF` → `GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf`
(arch `glm-dsa`, `block_count` 79, `expert_count` 256)
## Repro — single node, SSD streaming (fails)
```
./ds4 -m GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf --ssd-streaming -c 8192 -p "..."
```
Log:
```
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: SSD streaming mixed-precision model: 1/76 routed layers off the slab size class
will bypass the expert cache and read experts via mapped model views
ds4: GLM graph using compact DSA KV only; expanded full-attention KV cache is skipped
ds4: GLM graph allocating compact DSA cache: rows=8192 logical_ctx=8192 kv_layers=78 indexer_layers=21 f32 1.45 GiB
ds4: GLM compact indexed prefill chunk=4096 score_rows=4096 score_scratch=128.00 MiB
ds4: prompt processing failed: cuda GLM prefill failed
```
## Data point — 2-node pipeline, resident weights, CUDA (reaches prefill)
Same GGUF, split across two GB10 nodes with resident weights (no `--ssd-streaming`):
```
# worker (node B)
./ds4 -m --role worker --layers 40:output --coordinator 9911 -c 4096
# coordinator (node A)
./ds4 -m --role coordinator --layers 0:39 --listen 9911 -c 4096
```
This **does** load both slices (95.99 + 97.34 GiB), initialize the CUDA backend on both, allocate the GLM-DSA graph and reach prefill. It is then only blocked by memory (196 GiB model / 2×121 GiB nodes; the prefill working set spikes and leaves ~4 GiB headroom, so it gets externally killed to avoid a wedge). So the **resident** CUDA GLM-DSA prefill path appears functional — the failure above seems specific to the `--ssd-streaming` path.
## Questions
1. Is GLM `--ssd-streaming` prefill supported on the **CUDA** backend, or Metal/ROCm only? The surrounding code and env vars (`DS4_METAL_*`, `metal_graph_seed_streaming_expert_cache_from_hotlist`) look Metal/ROCm-oriented.
2. Is there a flag to get GLM-5.2 running single-node streamed on GB10, or is resident (multi-node) the only CUDA route for a model larger than one node's memory?
3. Cross-node `--tensor-parallel` returns `tensor parallelism requires the Metal backend`. Is CUDA cross-node TP planned, or is pipeline (`--role`/`--layers`) the intended CUDA multi-node path?
Thanks for the project — happy to run more diagnostics on GB10 if useful.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.