ml-explore / ml-explore/mlx

[BUG] JACCL segfaults in tbt_post_recv when a Thunderbolt cable is unplugged mid-collective

Open
#4,192 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug distributed
Dominant language
C++
Stars
28.5k
Forks
2.3k
Avg merge
3d 8h
Merged PRs (30d)
62

Description

Describe the bug

Unplugging the Thunderbolt cable during a JACCL collective crashes both ranks with SIGSEGV inside libthunderboltrdma.dylib, reached from jaccl::Connection::post_recv. Both ends die even though only one end is physically unplugged. No exception, no log line, process simply goes away.

This is distinct from #3910, which is a silent hang on peer loss. On this hardware, process death produces that hang; link loss produces this crash.

EXC_BAD_ACCESS  SIGSEGV  KERN_INVALID_ADDRESS at 0x0000000104ac4008
libthunderboltrdma.dylib  tbt_post_recv
libthunderboltrdma.dylib  tbt_post_recv
jaccl::Connection::post_recv(jaccl::SharedBuffer const&, unsigned long long)
void jaccl::MeshImpl::all_reduce<float, jaccl::SumOp<float>>(float const*, float*, long long, jaccl::SumOp<float>)
main

The faulting address is a few pages into a mapped region, consistent with a registered buffer whose backing went away with the device.

To Reproduce

Two Macs direct-attached over Thunderbolt, rdma_ctl enable done from Recovery OS on both, and an IPv4 address on each TB port (otherwise you hit #3467 first).

Two ranks running a collective in a loop:

auto g = jaccl::init();
float in[256], out[256];
for (int i = 0; i < 256; i++) in[i] = 1.0f;
for (;;) g->all_sum(in, out, sizeof(in), jaccl::Float32);

Unplug the cable at either end mid-run. Both ranks die immediately. 4/4 across two independent pulls, on unmodified MLX at 596dc79f4.

The last completed iteration takes ~200 ms versus ~3 us steady state, so something is noticed, but isn't survivable.

Expected behavior

Losing a link should surface as an error the caller can catch, the way a lost peer should. A SIGSEGV inside the driver gives the application nothing to handle and leaves no diagnostic.

I can't tell from outside whether the defect is that post_recv is called against a device that has gone away, or something in libthunderboltrdma itself. Happy to dig further if it's useful.

Desktop

  • 2x Mac mini M4 Pro (Mac16,11), macOS 26.5.1
  • Direct Thunderbolt 5, 80 Gb/s, RDMA enabled from Recovery OS
  • MLX at 596dc79f4, jaccl built from mlx/distributed/jaccl/lib

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in mlx/distributed/jaccl/lib by tracing jaccl::Connection::post_recv and its use from jaccl::MeshImpl::all_reduce. Reproduce with the two-Mac Thunderbolt setup and cable removal during the loop, then determine whether the failure is in JACCL or libthunderboltrdma. Done means link loss no longer causes SIGSEGV and instead surfaces an error the caller can catch, with diagnostic coverage for the regression.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, macos
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.