ml-explore / ml-explore/mlx

[BUG] JACCL segfaults in mx.distributed.init(backend="jaccl") on 2-node Mac-Mac with RDMA devices present

Open
#4,485 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
28.5k
Forks
2.3k
Avg merge
3d 8h
Merged PRs (30d)
62

Description

Description

mx.distributed.init(backend="jaccl") segfaults immediately (signal 11) during
2-node Mac↔Mac initialization, even when both hosts have working RDMA devices
(ibv_devices enumerates rdma_en2) and Thunderbolt 40 Gb/s links confirmed.
The segfault is 100% reproducible on fresh builds.

This differs from #3777 (closed by #4180) which covered the no-verbs-device
case. In our case, RDMA devices ARE present — the crash happens deeper in the
initialization path.

The same 2-node setup works fine with MlxRing (socket-based backend).

Environment

  • Hardware: 2× Apple Silicon Macs, 16 GB RAM, connected by direct Thunderbolt cable (40 Gb/s confirmed via system_profiler SPThunderboltDataType)
  • macOS: 26.6.2 (latest)
  • MLX: 0.32.0.dev20260806+cc3f3e60 (from rltakashige/mlx-jaccl-fix-small-recv fork, branch address-rdma-gpu-locks, commit cc3f3e60)
  • Both hosts verified:
    • rdma_ctl statusenabled
    • ibv_devicesrdma_en2 present (real verbs devices)
    • Thunderbolt link: active, 40 Gb/s

Steps to Reproduce

  1. Connect two Apple Silicon Macs via Thunderbolt cable
  2. Enable RDMA on both hosts (rdma_ctl enable, cold boot if needed)
  3. Verify ibv_devices shows rdma_en2 on both hosts
  4. Run mx.distributed.init(backend="jaccl") with both hosts in the group

Expected Behavior

JACCL initialization completes, allowing distributed operations over Thunderbolt RDMA.

Actual Behavior

Segfault immediately during initialization:

Fatal Python error: Segmentation fault
signal=11 (SIGSEGV)

Stack trace points into libjaccl.dylib — the shared-buffer / protection-domain
registration path during mx.distributed.init(backend="jaccl").

Isolation Evidence

Test Result
2-node MlxJaccl (RDMA), min_nodes: 2 Segfault signal 11 on both ranks
2-node MlxRing (socket), min_nodes: 2 Works — round-trip inference confirmed
Single-node on each Mac Works

The crash is isolated to JACCL initialization over RDMA. The Ring backend
uses the same topology and works. This isolates the bug to the JACCL/RDMA
initialization path specifically.

Related

  • Continuation of discussion in #3777 (comment by @cdvankammen, 2026-08-07) — maintainer @erwinzhang7 confirmed our failure is distinct from the #4180 fix
  • MlxRing (non-RDMA socket backend) works as a workaround

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the two-node setup with mx.distributed.init(backend="jaccl") and inspect the libjaccl.dylib shared-buffer and protection-domain registration path named in the report. Compare initialization with the working MlxRing backend and the single-node case. Done means JACCL initializes without a segmentation fault when RDMA devices are present and distributed operations can run.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, python
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.