PRO Q4 on two M3 Ultra 512GB: decode ~3.6 s/token on BOTH halves (~0.14 t/s); a single decode token is slower than a 512-token prefill

## Summary

Running the split DeepSeek V4 PRO Q4 GGUF across two Mac Studio M3 Ultra 512GB machines (coordinator 0:30, worker 31:output), the distributed route builds fine and text is coherent, but generation runs at 0.14 t/s (gen_tps=0.14, ~7.1 s/token) versus the README's 11.47 t/s smoke test — ~80x slower.

Both halves are symmetrically slow. The remote worker (layers 31:60) reports ~3.64 s/token, and by subtraction the coordinator's local layers 0:30 is ~3.5 s/token (7.1 s total − 3.64 s remote). Since the coordinator's local half never touches the network, this rules out the interconnect and the distributed protocol entirely — the problem is PRO Q4 per-token decode compute on the M3 Ultra.

The smoking gun: decoding 1 token (~7.1 s) takes longer than prefilling 512 tokens (~6.3 s, prefill_tps=80.71). For a 6-of-384 routed MoE this is impossible unless the sparse-expert fast path (the README's "queue-resident exact expert tables") is inactive during decode and each token re-binds the broad multi-GiB routed tensors. Each machine maps ~420 GB resident (426 / 411 GB) on a 512 GB box, leaving little headroom for the resident expert tables — consistent with the README's warning about hitting "Metal memory accounting limits."

## Environment

- Hardware: 2x Mac Studio Mac15,14, Apple M3 Ultra (32-core CPU: 24P+8E / 80-core GPU), 512 GB unified memory (one 2TB SSD, one 4TB SSD — no perf difference)
- OS: macOS 26.4.1 (build 25E253), system firmware 18000.101.7; built with Command Line Tools (Xcode.app not installed)
- Commit: 80ebbc396aee40eedc1d829222f3362d10fa4c6c (current main HEAD, 2026-06-17) — confirmed identical on both machines (coordinator and worker both report 80ebbc39)
- Model: DeepSeek-V4-Pro-Q4K-Layers00-30.gguf (coordinator) + DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf (worker)
- Link: direct Thunderbolt bridge, 192.168.2.1 to 192.168.2.2 (static IPs), ping 0.5–0.9 ms — matches the README's Thunderbolt reference (~0.45 ms), so the interconnect is not the bottleneck

## Repro

Worker (machine B):

```
./ds4 \
-m gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf \
--role worker \
--layers 31:output \
--coordinator 192.168.2.1 1234 \
--debug
```

Coordinator (machine A):

```
./ds4-bench \
-m gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf \
--prompt-file /tmp/smoke_512.txt \
--ctx-start 512 --ctx-max 512 --step-incr 512 \
--gen-tokens 8 \
--role coordinator \
--layers 0:30 \
--listen 192.168.2.1 1234 \
--dist-activation-bits 16 \
--debug
```

## Coordinator telemetry (the smoking gun)

```
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,kvcache_bytes
request=1 layers=31:60 pos=0 tokens=512 eval=3903.008ms downstream_wait=0.000ms forward_send=0.000ms input=28.00MiB
request=2 layers=31:60 pos=512 tokens=1 eval=3660.819ms downstream_wait=0.000ms forward_send=0.000ms input=0.05MiB
request=3 layers=31:60 pos=513 tokens=1 eval=3665.961ms downstream_wait=0.000ms forward_send=0.000ms input=0.05MiB
request=9 layers=31:60 pos=519 tokens=1 eval=3597.725ms downstream_wait=0.000ms forward_send=0.000ms input=0.05MiB
512,512,80.71,8,0.14,0
```

Decomposition of the ~7.1 s/token (from gen_tps=0.14):

| segment | per-token | where measured |
| --- | --- | --- |
| remote worker layers 31:60 | ~3.64 s | reported in telemetry |
| local coordinator layers 0:30 | ~3.5 s | = 7.1 s total − 3.64 s remote |
| network (downstream_wait + forward_send) | ~0 ms | telemetry |

Key observations:

- Both halves are ~3.5 s/token, symmetrically. The local half runs entirely on the coordinator with no network involved, so this is a compute problem, not a distributed/interconnect problem.
- Decode is slower than prefill of 512 tokens: 1 decode token ~7.1 s vs 512-token prefill ~6.3 s (prefill_tps=80.71). This indicates the sparse-MoE fast path is inactive during decode and each token re-binds/reads the broad routed tensors. Per half, ~420 GB / 3.5 s ≈ 120 GB/s effective, ~1/7 of the M3 Ultra's ~819 GB/s.

## What I've ruled out

| Hypothesis | Evidence | Verdict |
| --- | --- | --- |
| Network latency | Thunderbolt bridge, ping 0.5–0.9 ms; downstream_wait=0, forward_send=0 | ruled out |
| Network bandwidth | --dist-activation-bits 16 halved bytes, no change in 3.6 s/token | ruled out |
| CPU fallback / Metal not initialized | worker init log shows Metal residency + warmup + metal backend | ruled out |
| This machine is slow / memory pressure | single-machine Flash Q4 on same box is healthy (33 t/s) | ruled out |
| SSD size | 2TB and 4TB boxes benchmark within 0.2–0.3% | ruled out |

Single-machine Flash Q4 on the same hardware is normal:

```
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,kvcache_bytes
512,512,396.00,64,33.55,31042956
32768,32768,474.16,256,25.11,475014540
65536,32768,395.96,256,23.98,926033292
```

## Hypothesis

PRO Q4 decode isn't using the "queue-resident exact expert tables" path and instead re-binds/reads the broad routed tensors every token (the README's known slow/early failure mode). This happens on both halves, so it is not distributed-specific. Most likely trigger: each machine maps ~420 GB resident on a 512 GB box, leaving too little headroom for the resident expert tables, so Metal memory-accounting forces the fallback. Prefill (which reads broadly anyway, once per chunk) is only mildly affected; decode — where the sparse-gather fast path should dominate — pays the broad-binding cost per token and ends up slower than a 512-token prefill.

## Questions

- Is the "queue-resident exact expert tables" decode path expected to be active for split PRO Q4 on 2x 512 GB (each half ~420 GB resident), or does that headroom already trip the "Metal memory accounting limits" fallback the README mentions?
- Is there a memory-headroom threshold for the fast path? Would freeing more RAM, or a more headroom-preserving split, restore it?
- Any flag to force/verify the fast path, or extra telemetry (e.g. a per-token line for the coordinator's local layers) to confirm which decode path each half takes?

## Diagnostics (note: --trace does not apply here)

**--trace is not the right diagnostic for this bug.** --trace is a ds4-eval / ds4-server flag (defined in ds4_eval.c). ds4-bench rejects it, and plain ds4 produces no trace file for this run path. It also logs agent-session prompts / cache decisions / tool-parser events, not decode timing or Metal residency. The evidence for this performance bug is already inline above: the --debug per-hop telemetry, the ds4-bench CSV (gen_tps=0.14 / prefill_tps=80.71), and the memory_pressure wired-residency snapshots.

**Optional extra evidence, easy to capture:** the per-token Metal graph timing that --debug prints (same shape as #465, e.g. ds4: metal graph token pos=N encode=... execute=... total=...), and a memory_pressure snapshot taken DURING the slow decode on both machines to show the wired collapse with Swapouts=0.

```
# run on BOTH machines mid-decode to show wired collapse + zero swap:
memory_pressure
# per-token Metal timing already emitted by --debug (grep the run log):
# ds4: metal graph token pos=N encode=... execute=... total=...
```

---

## UPDATE — Root cause confirmed: wired residency collapses every token, on BOTH halves

Captured memory_pressure during an actual slow run on both machines. The wired (pinned/GPU-resident) footprint is NOT stable across the generation loop: it is wired up for each request's compute and released again — on BOTH the coordinator and the worker. This is Metal residency churn in physical RAM, not OS paging: Swapins/Swapouts stay 0 and Pageins increase by only ~34–44 pages between phases, so the hundreds of GiB that leave "wired" stay resident in RAM (as "inactive") and never touch SSD.

| machine / phase | wired | free % | swapouts |
| --- | --- | --- | --- |
| coordinator — prefill (request=1) | ~433 GiB | 15% | 0 |
| coordinator — decode (request>1) | ~38 GiB | 92% | 0 |
| worker — compute phase | ~232 GiB | 54% | 0 |
| worker — released phase | ~5.5 GiB | 98% | 0 |

The coordinator wires ~433 GiB (nearly its full 426 GiB local half) during compute, then drops to ~38 GiB; the worker peaks at only ~232 GiB (well below its 411 GiB half) and drops to ~5.5 GiB. Both oscillate per request. This directly explains: (a) decode being slower than a 512-token prefill (prefill wires once for 512 tokens; decode re-wires per token), and (b) both halves being symmetrically slow.

**Upgraded hypothesis (supersedes the section above):** during decode the PRO Q4 path does not keep the routed expert tables queue-resident; it releases their GPU/Metal residency after each request and re-binds the broad multi-GiB routed tensors for every token — the exact slow path the README's "queue-resident exact expert tables" was meant to avoid. The cost is residency re-establishment, not disk I/O (confirmed: Swapouts=0, Pageins flat). Whatever keeps the expert tables resident across the prefill call is not being kept alive across the per-token decode calls.

**Added ruled-out row:** OS memory pressure / swap — memory_pressure shows Swapins/Swapouts = 0 throughout and compressor ~8 MB (coordinator); the slowdown is not OS paging.

---

## Related issues / prior art (checked before filing)

I searched the README and existing issues. This scenario is not a duplicate — the closest reports are either single-machine, SSD-streaming, CUDA, or a different trigger. Mapping:

| # | relation | why it differs from this report |
| --- | --- | --- |
| #465 Unexpected slow decode when using --power | closest symptom: multi-second per-token Metal execute, ~0.24 t/s | single machine, triggered by --power, reporter suspects macOS 27 Beta GPU ramping, warmup mitigates |
| #455 Metal SSD streaming: exact path misses routed expert views | same subsystem: "Metal cannot wrap routed-expert model ranges" | SSD-streaming test path, not distributed in-RAM residency |
| #491 Expert bundle sidecar for contiguous MoE expert reads | same pain point (routed expert reads) | targets SSD-streaming cold-miss disk I/O; here there is no disk I/O |
| #437 / #439 Metal SSD-streaming benchmarks / cache pressure | same area (Metal routed-expert residency) | SSD streaming (model does not fit) |
| #293 Distributed worker host-registers the whole GGUF | distributed + whole-model registration concept | CUDA / DGX Spark, not Metal |
| #477 page-cache residency when RAM ≥ model (decode +52%) | residency optimization | CUDA, not Metal |
| README (queue-resident exact expert tables) | names the exact mechanism and the "Metal memory accounting limits" failure mode; flags distributed as not release-stable | documents it as a known hard area, but not this fits-in-RAM per-token residency collapse |

**What makes this report distinct (and, I think, a clean repro):**

- The model fully fits and stays in RAM — this is NOT SSD streaming. Wired peaks at ~433 GiB (coordinator) / ~232 GiB (worker), Swapouts stay 0, Pageins are flat across the prefill→decode transition. No disk I/O is involved.
- The wired residency of the routed expert tables collapses in RAM between prefill and decode and is re-established per token, on BOTH halves symmetrically.
- macOS 26.4.1 (not the 27 Beta suspected in #465); no --power flag; the slowdown is persistent across all decode tokens, not a first-token/warmup artifact.

---

## Additional evidence: the slowdown is sustained (rules out warmup; distinguishes from #465)

A gen_tokens=256 run shows 24 consecutive decode tokens all at ~3.6 s (eval 3565–3774 ms) — steady state, not a first-token/warmup artifact. This is the key difference from #465, where a warmup period mitigated the slowness. Two independent gen_tokens=8 runs reproduce the CSV: 80.71/0.14 and 81.40/0.14. Precise ping on the Thunderbolt bridge: 0.209 ms avg to 192.168.2.1, 0.762 ms avg to 192.168.2.2.

Full raw logs (ping, per-token telemetry for the 256- and 8-token runs, and both memory_pressure captures): https://gist.github.com/treemanz/271e62fe85dd3f9cfd815e997a917f63

Start by reproducing the two-machine ds4 worker/coordinator commands with --debug, then compare the per-token Metal graph timing, CSV throughput, and memory_pressure snapshots described here. Read the README's queue-resident exact expert tables and Metal memory-accounting notes; ds4_eval.c defines --trace, but the report says it does not apply to this path. Done means the decode path no longer shows per-token residency churn and the benchmark demonstrates sustained improvement.

Escrito por el modelo de indexación a partir del texto del issue.