[BUG] JACCL SIGSEGV in tbt_post_recv during unique all_sum (no unplug, no model, ports still up)

JACCL SIGSEGV in tbt_post_recv during unique all_sum (no unplug, no model, ports still up)

Related to #4192 (same tbt_post_* / …008 family) but not an unplug repro. The mesh stayed PORT_ACTIVE on every node.

Environment

• 4× Mac Studio M3 Ultra 512 GB, Thunderbolt 5 full mesh (6 edges)
• macOS 26.6.1, mlx 0.32.0, MLX_METAL_FAST_SYNCH=0
• JACCL over Thunderbolt RDMA. One mlx.distributed_config --auto-setup this boot, then this process only.

What we ran

No model. No all_gather on the hot path. Every 60 s:

• 256 MiB uint8
• exactly one rank injects a deterministic unique buffer; the other three inject zeros
• mx.distributed.all_sum + mx.eval + mx.synchronize
• every rank checks md5(result) == md5(unique) locally
• source rank rotates 0→1→2→3

This is the prefill collective (all_sum → JACCL all_reduce), not the 1 MiB unique all_gather used as a serve keepalive.

Result

Seven cycles OK (src 0,1,2,3,0,1,2). md5 matched on every rank. Then the next cycle died.

• last OK: cycle 7, src=2, 0.056 s
• ~62 s later (the next all_sum): rank 1 exit 255
• mlx.launch printed exit 0 (do not trust that)
• all four nodes still 3 PORT_ACTIVE

IPS on the rank-1 node (EXC_BAD_ACCESS / SIGSEGV):

KERN_INVALID_ADDRESS at 0x…008
libthunderboltrdma.dylib   tbt_post_recv
libthunderboltrdma.dylib   tbt_post_recv
libjaccl.dylib             jaccl::Connection::post_recv
libjaccl.dylib             jaccl::RingImpl::all_reduce<2, unsigned char, SumOp>
libmlx.dylib               mlx::core::scheduler::StreamThread::thread_fn

So this is RingImpl all_reduce on unsigned char, not MeshImpl::all_gather, and not bf16.

Why this is useful

The same tbt_post_recv / trailing-008 fault has shown up here on real TP4 prefills (RingImpl bf16 all_sum) and on serve keepalives (MeshImpl::all_gather). Those runs have a 1.65 T model and Metal in the picture. This one does not. Seven unique 256 MiB reductions succeeded, then the eighth died with the mesh still up.

Happy to share the ~80-line repro script and a redacted IPS. No patch attached. No claim this is a JACCL logic bug versus a closed libthunderboltrdma use-after-free; the IPS names Apple's dylib first.

Also still true

• mlx.launch exit 0 on rank 255 is a lie.
• A 1 MiB unique all_gather keepalive does not certify this path.
• Do not run distributed_config --auto-setup twice in one boot after a death; reboot the fleet.

Start with the reported IPS path through tbt_post_recv, jaccl::Connection::post_recv, and jaccl::RingImpl::all_reduce, then reproduce the rotating-source 256 MiB uint8 all_sum across the four-node mesh. Done would mean isolating whether the failure is in JACCL or libthunderboltrdma and documenting a verified correction or minimal reproducer.