Distributed only utilizing the GPU on the host machine

I have been playing with distributed training and generation and am running into a weird issue where only the computer that runs mpirun actually utilizes the GPU. The rest load the models into memory but do not actually use any GPU.

The command I am using to run the training is:

mpirun \
    --mca btl_tcp_links 4 \
    --mca btl_tcp_if_include bridge0 \
    --verbose \
    --mca btl_base_verbose 100 \
    -np 4 \
    -x PATH=$(pwd)/.venv/bin:$PATH \
    -x PYTHONPATH=$(pwd)/.venv/lib/python3.12/site-packages  \
    --hostfile ./hosts.txt \
    /opt/homebrew/bin/uv run mlx_lm.lora \
        --model mlx-community/Llama-3.3-70B-Instruct-4bit \
        --data ./converted \
        --train \
        --iters 1000 \
        --batch-size 1

I have tried not using thunderbolt etc and still the same issue. If I run the mpirun from any of my hosts I get the host that I start the program on using it's gpu but no other computer does.

my hosts.txt is pretty simple, ssh works, everything connects and training runs it just runs at 30 tokens / second and no distribution.

ai-mac-1.local  slots=1
ai-mac-2.local  slots=1
ai-mac-3.local  slots=1
ai-mac-4.local  slots=1

Start by reproducing the distributed run through the mlx_lm.lora entry point with the provided hosts.txt and mpirun command. Compare GPU utilization across all four hosts and inspect how the distributed training configuration is initialized. Done means each host loads and uses its GPU during training, rather than only the mpirun host.