antirez / antirez/ds4

SIGSEGV in ds4_gpu_encode_get_rows_q8_0 when running GLM-5.3-Flash distributed (pipeline parallelism) — crash at first generation step

Ouverte
#901 4 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
C
Étoiles
22.3k
Forks
2.1k
Merge moyen
1 j 3 h
PR mergées (30 j)
4

Description

**Title: SIGSEGV in ds4_gpu_encode_get_rows_q8_0 when running GLM-5.3-Flash distributed (pipeline parallelism) — crash at first generation step**

## Summary

GLM-5.3-Flash Q2 runs through distributed pipeline prefill successfully, but the coordinator deterministically crashes (EXC_BAD_ACCESS) as soon as generation begins. Reproduced 3× with identical stack. Prefill itself completes without errors and the route handshake is clean, so the failure is specific to the GLM distributed generation path.

## Environment

- Branch `glm-5.3-flash` @ `767e517` (both machines, rebuilt)
- 2× MacBook Pro M1 Max 64GB, macOS 26.6.2, arm64, Metal backend
- `glm53-q2` GGUF (~90 GiB, SHA256 `e81fd6241c6e55a64e1e14e47a3eab61a173fa8d7e4b5c1d1848827119705b32`), same file on both machines (verified after copy)
- Connected point-to-point over a Thunderbolt 4 bridge (static IPs, 0.8 ms RTT)

## Reproduction

Worker (machine B):

```sh
./ds4 --metal -m ~/models/GLM-5.3-Flash-Q2.gguf \
--role worker --layers 22:output \
--coordinator 192.168.100.1 5555 -c 16384
```

Coordinator (machine A):

```sh
./ds4 --metal -m /path/to/GLM-5.3-Flash-Q2.gguf \
--role coordinator --layers 0:21 \
--listen 192.168.100.1 5555 -c 16384 -n 48 --nothink \
-p "Introduce yourself in one sentence"
```

Route comes up (`local 0:21 -> Q2 22:output`), the 16-token prompt prefills to 100%, then the coordinator dies at the first generation step. Also reproduced with `-c 32768`, with `--debug`, and without `--nothink`.

## Crash (coordinator, 3/3 identical)

```
EXC_BAD_ACCESS (SIGSEGV), KERN_INVALID_ADDRESS at 0x3d59c1dfb8d2f2e4
(note: "possible pointer authentication failure")

Metal MTLResourceListAddResource
AGXMetalG13X -[AGXG13XFamilyComputeContext setComputePipelineState:]
ds4 ds4_gpu_encode_get_rows_q8_0
ds4 ds4_gpu_embed_tokens_q8_0_tensor
ds4 glm_graph_forward_indexed_tokens
ds4 ds4_session_eval_layer_slice
ds4 dist_coordinator_eval_span
ds4 ds4_dist_session_eval
```

A second variant crash (different run) aborted in `AutoreleasePoolPage::busted` with "autorelease pool page corrupted", which suggests the buffer holding the token/row arguments may be overrun slightly earlier, on the same code path (`glm_graph_forward_indexed_tokens` → `ds4_gpu_embed_tokens_q8_0_tensor`).

## Variables already ruled out

- Memory pressure (reproduced on a freshly-freed machine, 92% free)
- `--debug` telemetry (on/off)
- Context size (16384 / 32768)
- The worker side: loads and maps its slice fine (47.57 GiB resident, layers 22:44+output), stays healthy through the coordinator's crash

## Guess

`ds4_gpu_encode_get_rows_q8_0` is shared with the DeepSeek path; the GLM 5.3 embedding/n-gram table geometry (row size or alignment for the indexed get-rows) may differ in a way that leaves an argument buffer undersized or an unregistered MTLResource, hence the wild pointer when the encoder adds resources.

Happy to test a fix or provide more diagnostics (`DS4_DIST_CONNECT_TRACE`, full `.ips` files, worker-side logs).

---

**Disclosure:** This issue was drafted by GLM-5.3-Flash (running as the coding agent inside ZCode) at the direction of the machine owner, who reviewed and approved it. All crash reports, stack traces, and reproduction details above were collected first-hand on the owner's hardware.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.