[BUG] JACCL MeshImpl::recv spins forever on peer loss — silent hang, no timeout (UC)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 28.5k
- Forks
- 2.3k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 62
Description
Describe the bug
When a peer of an established JACCL mesh becomes unreachable mid-operation
(e.g. a Thunderbolt link drops), jaccl::MeshImpl::recv never returns: it
polls the dead QP forever. Because the transport is UC with no timeout and no
liveness check, the collective never completes, mx.eval blocks forever on
Event::wait, and the whole distributed job hangs with zero errno, zero
exception, zero log output. Processes stay alive; SIGTERM is never acted on
(the interpreter is blocked inside native code), so the only way out is
SIGKILL — which on macOS leaves the Metal/RDMA wired pages kernel-held until a
reboot.
This is distinct from the connect-time failures (#3467, #3755): init succeeded,
traffic was flowing, and the hang appears only when a peer goes away under an
established QP.
Native stack of a hung rank (captured live with sample, mlx 0.32.0)
Main thread (why SIGTERM is never delivered):
_PyEval_EvalFrameDefault
mlx::core::eval(std::vector<mlx::core::array>)
mlx::core::array::wait()
mlx::core::Event::wait()
-[IOSurfaceSharedEvent waitUntilSignaledValue:timeoutMS:]
StreamThread (the actual spin — 1156/1304 samples inside MeshImpl::recv):
mlx::core::scheduler::StreamThread::thread_fn()
jaccl::MeshImpl::recv(char*, long long, int) libjaccl.dylib
tbt_poll_cq libthunderboltrdma.dylib
tbt_poll_qp_recv → ring_indicies_err (error path polled in a loop)
tbt_poll_qp_recv → ring_get_completions_to_check
tbt_poll_qp_send → ring_indicies_err
Note the driver's poll path is repeatedly hitting ring_indicies_err — the
information that the QP is in trouble appears to be visible at the driver
level, but MeshImpl::recv keeps polling regardless.
To Reproduce (deterministic)
- 2+ Macs, TB5 mesh,
backend="jaccl"(MeshGroup), any workload doing
point-to-pointsend/recvor collectives in a loop (pipeline-parallel
inference is a natural one). - Mid-operation, unplug the TB cable between two nodes.
- All ranks freeze silently. Re-plugging the cable ~1 min later did not
unblock the in-flight operation in our test (the QP appears dead for good;
only re-init recovers).
Additional occurrence (trigger not captured)
We also observed what appears to be the same signature after ~75 h of
continuous serving with no physical action. That trigger was not captured and
may have a different root cause; reporting it only as context for how this
class of hang shows up in long-running deployments.
Expected behavior
Either of:
recv/collectives fail with an error when the peer/link is gone (the driver
seems to know — seering_indicies_errabove), or- an optional bounded-poll / timeout knob (env var) so integrators can turn a
silent hang into a catchable failure.
Today the only robust integration is an application-level no-progress watchdog
that SIGKILLs the ranks and reboots the leaked nodes, which is heavy.
Desktop
- 4× Mac Studio M3 Ultra (Mac15,14), macOS 26.4.1 and 26.5 (mixed across nodes)
- mlx 0.32.0 (pip), python 3.11
- TB5 full mesh, RDMA enabled; init is clean (this is not the #3467
GID-selection issue — we work around that one with static link-local IPv4
per TB port, see #3467)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at jaccl::MeshImpl::recv and trace how tbt_poll_qp_recv reports ring_indicies_err during an established-QP peer loss. Examine how the native path propagates transport failures into MLX's Event::wait; no test file is named. Done means a lost peer produces a catchable failure or a bounded timeout instead of indefinite polling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend-api-design, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100