ml-explore / ml-explore/mlx

[BUG] JACCL MeshImpl::recv spins forever on peer loss — silent hang, no timeout (UC)

Open
#3,910 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug distributed
Dominant language
C++
Stars
28.5k
Forks
2.3k
Avg merge
3d 8h
Merged PRs (30d)
62

Description

Describe the bug

When a peer of an established JACCL mesh becomes unreachable mid-operation
(e.g. a Thunderbolt link drops), jaccl::MeshImpl::recv never returns: it
polls the dead QP forever. Because the transport is UC with no timeout and no
liveness check, the collective never completes, mx.eval blocks forever on
Event::wait, and the whole distributed job hangs with zero errno, zero
exception, zero log output
. Processes stay alive; SIGTERM is never acted on
(the interpreter is blocked inside native code), so the only way out is
SIGKILL — which on macOS leaves the Metal/RDMA wired pages kernel-held until a
reboot.

This is distinct from the connect-time failures (#3467, #3755): init succeeded,
traffic was flowing, and the hang appears only when a peer goes away under an
established QP.

Native stack of a hung rank (captured live with sample, mlx 0.32.0)

Main thread (why SIGTERM is never delivered):

_PyEval_EvalFrameDefault
  mlx::core::eval(std::vector<mlx::core::array>)
    mlx::core::array::wait()
      mlx::core::Event::wait()
        -[IOSurfaceSharedEvent waitUntilSignaledValue:timeoutMS:]

StreamThread (the actual spin — 1156/1304 samples inside MeshImpl::recv):

mlx::core::scheduler::StreamThread::thread_fn()
  jaccl::MeshImpl::recv(char*, long long, int)          libjaccl.dylib
    tbt_poll_cq                                          libthunderboltrdma.dylib
      tbt_poll_qp_recv → ring_indicies_err               (error path polled in a loop)
      tbt_poll_qp_recv → ring_get_completions_to_check
      tbt_poll_qp_send → ring_indicies_err

Note the driver's poll path is repeatedly hitting ring_indicies_err — the
information that the QP is in trouble appears to be visible at the driver
level, but MeshImpl::recv keeps polling regardless.

To Reproduce (deterministic)

  • 2+ Macs, TB5 mesh, backend="jaccl" (MeshGroup), any workload doing
    point-to-point send/recv or collectives in a loop (pipeline-parallel
    inference is a natural one).
  • Mid-operation, unplug the TB cable between two nodes.
  • All ranks freeze silently. Re-plugging the cable ~1 min later did not
    unblock the in-flight operation in our test (the QP appears dead for good;
    only re-init recovers).

Additional occurrence (trigger not captured)

We also observed what appears to be the same signature after ~75 h of
continuous serving with no physical action. That trigger was not captured and
may have a different root cause; reporting it only as context for how this
class of hang shows up in long-running deployments.

Expected behavior

Either of:

  • recv/collectives fail with an error when the peer/link is gone (the driver
    seems to know — see ring_indicies_err above), or
  • an optional bounded-poll / timeout knob (env var) so integrators can turn a
    silent hang into a catchable failure.

Today the only robust integration is an application-level no-progress watchdog
that SIGKILLs the ranks and reboots the leaked nodes, which is heavy.

Desktop

  • 4× Mac Studio M3 Ultra (Mac15,14), macOS 26.4.1 and 26.5 (mixed across nodes)
  • mlx 0.32.0 (pip), python 3.11
  • TB5 full mesh, RDMA enabled; init is clean (this is not the #3467
    GID-selection issue — we work around that one with static link-local IPv4
    per TB port, see #3467)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at jaccl::MeshImpl::recv and trace how tbt_poll_qp_recv reports ring_indicies_err during an established-QP peer loss. Examine how the native path propagates transport failures into MLX's Event::wait; no test file is named. Done means a lost peer produces a catchable failure or a bounded timeout instead of indefinite polling.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend-api-design, distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.