antirez / antirez/ds4

GLM-5.2 (glm-dsa) prefill fails on CUDA / DGX Spark GB10 (sm_121): "cuda GLM prefill failed" with --ssd-streaming

Đang mở
#595 0 bình luận 1 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
C
Star
22.3k
Fork
2.1k
Merge trung bình
1 ngày 3 giờ
Pull request đã merge (30 ngày)
4

Mô tả

## Summary

On a DGX Spark (GB10, sm_121, CUDA 13.0, built with `make cuda-spark`), GLM-5.2 IQ2_XXS single-node with `--ssd-streaming` loads and allocates the GLM compact-DSA graph, but then fails at prefill:

```
ds4: prompt processing failed: cuda GLM prefill failed
```

There is no OOM (~116 GiB free at the time). DeepSeek-V4-Flash is the CUDA path this engine primarily targets and builds fine; this looks GLM-specific on the CUDA backend.

## Environment

- Hardware: NVIDIA DGX Spark / GB10 (sm_121), 128 GiB unified memory
- Build: `make cuda-spark`, nvcc CUDA 13.0, Ubuntu 24.04, driver 580.x
- Model: `antirez/GLM-5.2-GGUF` → `GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf`
(arch `glm-dsa`, `block_count` 79, `expert_count` 256)

## Repro — single node, SSD streaming (fails)

```
./ds4 -m GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf --ssd-streaming -c 8192 -p "..."
```

Log:

```
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: SSD streaming mixed-precision model: 1/76 routed layers off the slab size class
will bypass the expert cache and read experts via mapped model views
ds4: GLM graph using compact DSA KV only; expanded full-attention KV cache is skipped
ds4: GLM graph allocating compact DSA cache: rows=8192 logical_ctx=8192 kv_layers=78 indexer_layers=21 f32 1.45 GiB
ds4: GLM compact indexed prefill chunk=4096 score_rows=4096 score_scratch=128.00 MiB
ds4: prompt processing failed: cuda GLM prefill failed
```

## Data point — 2-node pipeline, resident weights, CUDA (reaches prefill)

Same GGUF, split across two GB10 nodes with resident weights (no `--ssd-streaming`):

```
# worker (node B)
./ds4 -m --role worker --layers 40:output --coordinator 9911 -c 4096
# coordinator (node A)
./ds4 -m --role coordinator --layers 0:39 --listen 9911 -c 4096
```

This **does** load both slices (95.99 + 97.34 GiB), initialize the CUDA backend on both, allocate the GLM-DSA graph and reach prefill. It is then only blocked by memory (196 GiB model / 2×121 GiB nodes; the prefill working set spikes and leaves ~4 GiB headroom, so it gets externally killed to avoid a wedge). So the **resident** CUDA GLM-DSA prefill path appears functional — the failure above seems specific to the `--ssd-streaming` path.

## Questions

1. Is GLM `--ssd-streaming` prefill supported on the **CUDA** backend, or Metal/ROCm only? The surrounding code and env vars (`DS4_METAL_*`, `metal_graph_seed_streaming_expert_cache_from_hotlist`) look Metal/ROCm-oriented.
2. Is there a flag to get GLM-5.2 running single-node streamed on GB10, or is resident (multi-node) the only CUDA route for a model larger than one node's memory?
3. Cross-node `--tensor-parallel` returns `tensor parallelism requires the Metal backend`. Is CUDA cross-node TP planned, or is pipeline (`--role`/`--layers`) the intended CUDA multi-node path?

Thanks for the project — happy to run more diagnostics on GB10 if useful.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.