Ring backend: recv fails with errno 14 (EFAULT) during multi-node pipeline inference, fabric-independent (distinct from #3862)

Summary

During multi-node pipeline inference over the ring backend (mx.distributed, backend="ring"), the receiving rank fails with errno 14 (EFAULT) on recv, repeats until the 10-error threshold, then aborts:

[ring] Receiving from socket 51 failed with errno 14
[ring] Receiving from socket 51 failed with errno 14
... (x10 within milliseconds)
[ring] Too many send/recv errors. Aborting...

The peer rank then logs errno 32 (EPIPE) echoes when the aborting side's destructor shuts the sockets.

We believe this is distinct from #3862: that report's trigger is errno 54 (ECONNRESET — a network-layer reset), while EFAULT from recv(2) indicates the buffer pointer passed to the syscall is invalid — a memory-side fault, not a transport drop. Consistent with that reading, the failure reproduces identically across different physical fabrics (1 GbE ethernet and a direct Thunderbolt bridge link), which a network-layer cause would not predict.

Environment

• mlx 0.32.0 (PyPI wheel), Python 3.13, macOS (Apple Silicon)
• Nodes tested in 2-node rings, all combinations of: Mac15,14 (96 GB), 2× Mac14,8 M2 Ultra (64 GB), Mac16,11 M4 Pro (48 GB)
• libmlx.dylib md5 identical on 3 of 4 machines; failures reproduced on pairs with identical hashes, so binary skew is not the trigger
• Invoked via exo v0.3.70's MlxRing pipeline runner (hostfile with 2 ranks); model mlx-community/Llama-3.3-70B-Instruct-4bit (fits each pair's memory comfortably; shards sized ~13–26 GB per rank)

Reproduction pattern (three controlled runs)

1. Pair A–B over 1 GbE: ranks init (Group size: 2), weights load, warmup starts → receiving rank EFAULT burst → abort.
2. Pair A–C over direct Thunderbolt bridge: identical signature — sender EPIPE + receiver EFAULT.
3. Pair B–D (excluding node A): got further — prefill completed across the ring (~1.8 s), decode started — then wedged silently: one rank pinned at ~100 % CPU in the ring wait loop with GPU at 0 MHz / 0 mW (powermetrics), no error lines. This quiet wedge matches the abandoned-promises behavior described in #3862 (and #4060's fix would at least convert it to a raised error — though we note #4060 is merged but not yet in any release).

After the abort, runners remain alive but stuck (pre-#4060 behavior), holding partially-loaded weights.

What we ruled out

• Memory pressure (placement sized under measured free memory; also reproduced with a model at ~35 % of aggregate RAM)
• Version skew (0.32.0 everywhere; PyPI has no newer release as of 2026-08-15)
• Reachability/firewall (all inter-rank TCP ports verified open both directions)
• A single bad machine (three different pairs, four machines, same class of failure)

Ask

Is EFAULT-on-recv a known failure mode of the ring backend on Metal? Candidate mechanisms we can't distinguish from outside: recv into a buffer whose Metal backing was freed/migrated mid-transfer, or an offset/pointer bug in the chunked recv path. Happy to run MLX_RING_VERBOSE=1, a minimal mlx.launch -n2 + all_sum repro, or a main-build test if that helps — the hardware and failure are readily available to us.

Start with the ring backend used through mx.distributed and reproduce the two-rank case using MLX_RING_VERBOSE=1 or a minimal mlx.launch -n2 plus all_sum test. Compare the EFAULT/EPIPE failure and silent wait-loop wedge across the reported fabrics, and test a main build including the #4060 change; done means the recv failure mechanism is identified and a reliable fix or narrowed reproducer is documented.