Hardware Bottleneck: AppleThunderboltRDMA hard limit of 100 MRs caps JACCL multi-node scaling

Hey folks,

While experimenting with multi-node Tensor Parallelism across a small Mac Studio cluster on macOS 26.3, we ran into what appears to be a memory region (MR) registration ceiling in the AppleThunderboltRDMA driver that seems to cap out around 100 MRs per device. We're not 100% sure if this is a hard driver limit or something specific to our setup, but we wanted to share what we've seen in case it's useful.

We put together a small libibverbs stress test (independent of MLX) to try to isolate the behavior, and here's what we observed across our 3 nodes:

What We Saw

Buffer size per MR: 512KB. Registrations used IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ.

Why It Might Matter for JACCL

If we're reading the JACCL code correctly, MR allocation scales roughly as: MRs ≈ BUFFER_SIZES × NUM_BUFFERS × size_ × 2 (send + recv). If that's right, and assuming BUFFER_SIZES=8, the math looks something like:

So it seems like the current NUM_BUFFERS=4 with a 2-node TP pair sits safely within limits, but there isn't much room to grow the pipeline depth without bumping into this ceiling.

Practical Impact Even at Small Scale (2–4 Nodes)

This isn't just a server-scale concern — even for small Thunderbolt-connected Mac Studio clusters (which max out at ~4 nodes physically), the MR limit constrains performance optimization:

• Pipeline depth = throughput. Increasing NUM_BUFFERS allows more RDMA transfers to be in-flight simultaneously, which helps overlap communication with computation. At NUM_BUFFERS=4 we're leaving performance on the table, but can't safely go higher without risking the MR ceiling.
• At 3 nodes, NUM_BUFFERS=4 already uses ~96 of ~100 MRs, leaving almost zero headroom. Any additional MR usage (e.g., from future features or optimizations) would push it over.
• At 4 nodes, even NUM_BUFFERS=3 would use ~96 MRs, meaning the pipeline would have to be made shallower as you add nodes — the opposite of what you'd want for performance.

Being able to deepen the pipeline (e.g., NUM_BUFFERS=8) at even 2 nodes could meaningfully improve throughput by better hiding RDMA latency behind GPU compute. Right now the limit makes that impossible.

Some Observations (Take With a Grain of Salt)

1. The limit being exactly 100 (a round decimal number) makes us think it might be a software-defined cap rather than a hardware constraint, but we could be wrong about that.
2. On older macOS builds, we saw segfaults when exceeding this limit; on 26.3 it seems to return NULL gracefully, which is a nice improvement.

The Question

We realize this may be outside the scope of the MLX project itself, but since the distributed backend is affected by this limit — would the team happen to know if there's a way to raise it, or if it'd be possible to flag this to the macOS kernel/I/O team? Even bumping it to 256 or making it configurable via sysctl could open up a lot of headroom for multi-node inference scaling. Totally understand if this isn't something you can action on — just wanted to surface the data point.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <infiniband/verbs.h>

#define BUFFER_SIZE (512 * 1024)
#define MAX_MRS 256

int main() {
    struct ibv_device **dev_list;
    struct ibv_context *context;
    struct ibv_pd *pd;
    int num_devices;

    dev_list = ibv_get_device_list(&num_devices);
    if (!dev_list || num_devices == 0) return 1;

    context = ibv_open_device(dev_list[0]);
    if (!context) return 1;

    pd = ibv_alloc_pd(context);
    if (!pd) return 1;

    struct ibv_mr *mrs[MAX_MRS];
    void *buffers[MAX_MRS];

    for (int i = 0; i < MAX_MRS; i++) {
        posix_memalign(&buffers[i], 4096, BUFFER_SIZE);
        memset(buffers[i], 0xAA, BUFFER_SIZE);

        mrs[i] = ibv_reg_mr(pd, buffers[i], BUFFER_SIZE, 
                            IBV_ACCESS_LOCAL_WRITE | 
                            IBV_ACCESS_REMOTE_WRITE | 
                            IBV_ACCESS_REMOTE_READ);

        if (!mrs[i]) {
            fprintf(stderr, "\n[!] ibv_reg_mr gracefully failed at iteration %d\n", i);
            break;
        }
    }
    return 0;
}

Start by running the provided rdma_stress.c reproduction on the affected macOS and AppleThunderboltRDMA setup, then compare its results with the JACCL MR allocation described in the issue. Done would require confirming whether the limit is imposed by the driver or setup and identifying an actionable path for MLX or an upstream macOS/kernel report.