antirez / antirez/ds4

GLM-5.2 (glm-dsa) prefill fails on CUDA / DGX Spark GB10 (sm_121): "cuda GLM prefill failed" with --ssd-streaming

Offen
#595 0 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C
Sterne
22.3k
Forks
2.1k
Ø Merge
1 T. 3 Std.
Gemergte PRs (30 T.)
4

Beschreibung

## Summary

On a DGX Spark (GB10, sm_121, CUDA 13.0, built with `make cuda-spark`), GLM-5.2 IQ2_XXS single-node with `--ssd-streaming` loads and allocates the GLM compact-DSA graph, but then fails at prefill:

```
ds4: prompt processing failed: cuda GLM prefill failed
```

There is no OOM (~116 GiB free at the time). DeepSeek-V4-Flash is the CUDA path this engine primarily targets and builds fine; this looks GLM-specific on the CUDA backend.

## Environment

- Hardware: NVIDIA DGX Spark / GB10 (sm_121), 128 GiB unified memory
- Build: `make cuda-spark`, nvcc CUDA 13.0, Ubuntu 24.04, driver 580.x
- Model: `antirez/GLM-5.2-GGUF` → `GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf`
(arch `glm-dsa`, `block_count` 79, `expert_count` 256)

## Repro — single node, SSD streaming (fails)

```
./ds4 -m GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf --ssd-streaming -c 8192 -p "..."
```

Log:

```
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: SSD streaming mixed-precision model: 1/76 routed layers off the slab size class
will bypass the expert cache and read experts via mapped model views
ds4: GLM graph using compact DSA KV only; expanded full-attention KV cache is skipped
ds4: GLM graph allocating compact DSA cache: rows=8192 logical_ctx=8192 kv_layers=78 indexer_layers=21 f32 1.45 GiB
ds4: GLM compact indexed prefill chunk=4096 score_rows=4096 score_scratch=128.00 MiB
ds4: prompt processing failed: cuda GLM prefill failed
```

## Data point — 2-node pipeline, resident weights, CUDA (reaches prefill)

Same GGUF, split across two GB10 nodes with resident weights (no `--ssd-streaming`):

```
# worker (node B)
./ds4 -m --role worker --layers 40:output --coordinator 9911 -c 4096
# coordinator (node A)
./ds4 -m --role coordinator --layers 0:39 --listen 9911 -c 4096
```

This **does** load both slices (95.99 + 97.34 GiB), initialize the CUDA backend on both, allocate the GLM-DSA graph and reach prefill. It is then only blocked by memory (196 GiB model / 2×121 GiB nodes; the prefill working set spikes and leaves ~4 GiB headroom, so it gets externally killed to avoid a wedge). So the **resident** CUDA GLM-DSA prefill path appears functional — the failure above seems specific to the `--ssd-streaming` path.

## Questions

1. Is GLM `--ssd-streaming` prefill supported on the **CUDA** backend, or Metal/ROCm only? The surrounding code and env vars (`DS4_METAL_*`, `metal_graph_seed_streaming_expert_cache_from_hotlist`) look Metal/ROCm-oriented.
2. Is there a flag to get GLM-5.2 running single-node streamed on GB10, or is resident (multi-node) the only CUDA route for a model larger than one node's memory?
3. Cross-node `--tensor-parallel` returns `tensor parallelism requires the Metal backend`. Is CUDA cross-node TP planned, or is pipeline (`--role`/`--layers`) the intended CUDA multi-node path?

Thanks for the project — happy to run more diagnostics on GB10 if useful.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.