antirez / antirez/ds4

GLM-5.2 (glm-dsa) prefill fails on CUDA / DGX Spark GB10 (sm_121): "cuda GLM prefill failed" with --ssd-streaming

オープン
#595 コメント 0 件 リアクション 1 件 担当者 0 名 GitHub で見る
主要言語
C
スター
22.3k
フォーク
2.1k
平均マージ
1日 3時間
マージ済み PR(30日)
4

説明

## Summary

On a DGX Spark (GB10, sm_121, CUDA 13.0, built with `make cuda-spark`), GLM-5.2 IQ2_XXS single-node with `--ssd-streaming` loads and allocates the GLM compact-DSA graph, but then fails at prefill:

```
ds4: prompt processing failed: cuda GLM prefill failed
```

There is no OOM (~116 GiB free at the time). DeepSeek-V4-Flash is the CUDA path this engine primarily targets and builds fine; this looks GLM-specific on the CUDA backend.

## Environment

- Hardware: NVIDIA DGX Spark / GB10 (sm_121), 128 GiB unified memory
- Build: `make cuda-spark`, nvcc CUDA 13.0, Ubuntu 24.04, driver 580.x
- Model: `antirez/GLM-5.2-GGUF` → `GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf`
(arch `glm-dsa`, `block_count` 79, `expert_count` 256)

## Repro — single node, SSD streaming (fails)

```
./ds4 -m GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf --ssd-streaming -c 8192 -p "..."
```

Log:

```
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: SSD streaming mixed-precision model: 1/76 routed layers off the slab size class
will bypass the expert cache and read experts via mapped model views
ds4: GLM graph using compact DSA KV only; expanded full-attention KV cache is skipped
ds4: GLM graph allocating compact DSA cache: rows=8192 logical_ctx=8192 kv_layers=78 indexer_layers=21 f32 1.45 GiB
ds4: GLM compact indexed prefill chunk=4096 score_rows=4096 score_scratch=128.00 MiB
ds4: prompt processing failed: cuda GLM prefill failed
```

## Data point — 2-node pipeline, resident weights, CUDA (reaches prefill)

Same GGUF, split across two GB10 nodes with resident weights (no `--ssd-streaming`):

```
# worker (node B)
./ds4 -m --role worker --layers 40:output --coordinator 9911 -c 4096
# coordinator (node A)
./ds4 -m --role coordinator --layers 0:39 --listen 9911 -c 4096
```

This **does** load both slices (95.99 + 97.34 GiB), initialize the CUDA backend on both, allocate the GLM-DSA graph and reach prefill. It is then only blocked by memory (196 GiB model / 2×121 GiB nodes; the prefill working set spikes and leaves ~4 GiB headroom, so it gets externally killed to avoid a wedge). So the **resident** CUDA GLM-DSA prefill path appears functional — the failure above seems specific to the `--ssd-streaming` path.

## Questions

1. Is GLM `--ssd-streaming` prefill supported on the **CUDA** backend, or Metal/ROCm only? The surrounding code and env vars (`DS4_METAL_*`, `metal_graph_seed_streaming_expert_cache_from_hotlist`) look Metal/ROCm-oriented.
2. Is there a flag to get GLM-5.2 running single-node streamed on GB10, or is resident (multi-node) the only CUDA route for a model larger than one node's memory?
3. Cross-node `--tensor-parallel` returns `tensor parallelism requires the Metal backend`. Is CUDA cross-node TP planned, or is pipeline (`--role`/`--layers`) the intended CUDA multi-node path?

Thanks for the project — happy to run more diagnostics on GB10 if useful.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。