[BUG] JACCL segfaults in tbt_post_recv when a Thunderbolt cable is unplugged mid-collective
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 28.5k
- Forks
- 2.3k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 62
Description
Describe the bug
Unplugging the Thunderbolt cable during a JACCL collective crashes both ranks with SIGSEGV inside libthunderboltrdma.dylib, reached from jaccl::Connection::post_recv. Both ends die even though only one end is physically unplugged. No exception, no log line, process simply goes away.
This is distinct from #3910, which is a silent hang on peer loss. On this hardware, process death produces that hang; link loss produces this crash.
EXC_BAD_ACCESS SIGSEGV KERN_INVALID_ADDRESS at 0x0000000104ac4008
libthunderboltrdma.dylib tbt_post_recv
libthunderboltrdma.dylib tbt_post_recv
jaccl::Connection::post_recv(jaccl::SharedBuffer const&, unsigned long long)
void jaccl::MeshImpl::all_reduce<float, jaccl::SumOp<float>>(float const*, float*, long long, jaccl::SumOp<float>)
main
The faulting address is a few pages into a mapped region, consistent with a registered buffer whose backing went away with the device.
To Reproduce
Two Macs direct-attached over Thunderbolt, rdma_ctl enable done from Recovery OS on both, and an IPv4 address on each TB port (otherwise you hit #3467 first).
Two ranks running a collective in a loop:
auto g = jaccl::init();
float in[256], out[256];
for (int i = 0; i < 256; i++) in[i] = 1.0f;
for (;;) g->all_sum(in, out, sizeof(in), jaccl::Float32);
Unplug the cable at either end mid-run. Both ranks die immediately. 4/4 across two independent pulls, on unmodified MLX at 596dc79f4.
The last completed iteration takes ~200 ms versus ~3 us steady state, so something is noticed, but isn't survivable.
Expected behavior
Losing a link should surface as an error the caller can catch, the way a lost peer should. A SIGSEGV inside the driver gives the application nothing to handle and leaves no diagnostic.
I can't tell from outside whether the defect is that post_recv is called against a device that has gone away, or something in libthunderboltrdma itself. Happy to dig further if it's useful.
Desktop
- 2x Mac mini M4 Pro (
Mac16,11), macOS 26.5.1 - Direct Thunderbolt 5, 80 Gb/s, RDMA enabled from Recovery OS
- MLX at
596dc79f4, jaccl built frommlx/distributed/jaccl/lib
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in mlx/distributed/jaccl/lib by tracing jaccl::Connection::post_recv and its use from jaccl::MeshImpl::all_reduce. Reproduce with the two-Mac Thunderbolt setup and cable removal during the loop, then determine whether the failure is in JACCL or libthunderboltrdma. Done means link loss no longer causes SIGSEGV and instead surfaces an error the caller can catch, with diagnostic coverage for the regression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, macos
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100