hellas-ai / hellas-ai/thunderbolt-ibverbs

Two-PC Linux setup: native HELLO/READY negotiates and registers usb4_rdma0/rocep* at kernel level, but ibverbs provider never claims it (no driver_id, PCI ID mismatch?)

Open
#74 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
60
Forks
6
PR merge metrics
No merged PRs in 30d

Description

## Setup

Two AMD Strix Halo boxes (Ryzen AI Max+ 395, 128GB unified memory each), connected directly via a USB4 cable between their native AMD USB4 host routers (`1022:158d` / `1022:158e`, "Strix/Strix Halo PCIe USB4 Bridge" + "Strix Halo USB4 Host Router"). Ubuntu 26.04, kernel `7.0.0-29-generic`. Goal: use this alongside `ds4`'s `--tensor-parallel --transport rdma`, since our other RDMA link (Mellanox ConnectX-3 Pro in native InfiniBand mode) doesn't expose an IPv4-mapped GID that `ds4`'s RDMA connection setup needs (LID-addressed fabric, not RoCE).

## What worked

- `thunderbolt-ibverbs-dkms_0.3.4_all.deb` builds and signs cleanly via DKMS against this kernel (needed Secure Boot MOK enrollment on both machines — worth a mention in the README, took us a couple of reboots to get the MOK password to register right at the pre-boot MOK Manager screen).
- The `usb4-rdma-provider_0.3.4.noble_amd64.deb` (Ubuntu 24.04 build) installs a provider `.so` tagged `-rdmav34`, but our system's `ibverbs-providers` (rdma-core 61.0, Ubuntu 26.04) expects ABI `v59` (every other provider in `/usr/lib/x86_64-linux-gnu/libibverbs/` is `-rdmav59.so`). `libibverbs` warns `couldn't load driver 'libusb4_rdma-rdmav59.so'` and silently skips it. The generic, non-codename `usb4-rdma-provider_0.3.4_amd64.deb` *does* ship `-rdmav59.so` and matches — might be worth making that the default recommendation for non-LTS/newer Ubuntu, or bumping the `noble` build's target ABI.
- Loading the module bare (`modprobe thunderbolt_ibverbs`) leaves `Thunderbolt service binding disabled` in dmesg and no negotiation happens. Had to pass:
```
modprobe thunderbolt_ibverbs profile=linux_perf bind_services=1 allocate_rings=1 start_rings=1 negotiate_native=1 enable_tunnels=1 register_verbs=1
```
on **both** ends before any HELLO/HELLO_ACK/READY exchange happened. This isn't mentioned in the top-level README (only found it by asking an assistant to dig through the repo) — might be worth calling out as required, not optional, for the two-PC-Linux case.

## What's still broken

With those params, both sides negotiate successfully:

```
thunderbolt_ibverbs: native HELLO negotiated route=0x2 rail=0x0 remote_out=9 remote_tx=2 remote_rx=2 attempt=1
thunderbolt_ibverbs: enabled tunnel route=0x2 rail=0x0 out_hop=9 remote_out_hop=9 tx_hop=2 rx_hop=2
thunderbolt_ibverbs: native READY received route=0x2 rail=0x0
thunderbolt_ibverbs: registered native ib_device usb4_rdma0 rail=peer1/0 domain=0 guid=0200544257524253
```

...on both machines, and a new device shows up in `/sys/class/infiniband/` (`rocep200s0f5`) plus a fresh `/dev/infiniband/uverbs1` + `umad2`. But `ibv_devices` / `ibv_devinfo` never list it — the provider library loads (`init` is called, confirmed via `LD_DEBUG=libs`) but doesn't claim the device.

Checked `/sys/class/infiniband_verbs/uverbs1/device/driver_id` — doesn't exist. Modern rdma-core prefers matching providers via that sysfs attribute; without it, it presumably falls back to a PCI vendor/device table baked into the provider `.so`. `uverbs1/device/modalias` reports `pci:v00001022d0000158Dsv00001022sd0000158Dbc0Csc03i40` (the AMD USB4 host router's real PCI ID) — worth checking whether the provider's fallback matching table actually includes `1022:158d`/`158e`, or expects something else (maybe a synthetic ID tied to the emulated `rocep*` device rather than the physical USB4 controller?).

Happy to grab more debug output (`RDMAV_HUGEPAGES_SAFE`, provider tracing, whatever's useful) if that'd help narrow it down — this got us most of the way there, just missing the last userspace hop.

There's also an unrelated `thunderbolt 0000:c8:00.5: invalid hop: -1` / `probe with driver thunderbolt_ibverbs failed with error -12` right before the successful native registration on both machines — looked benign (registration succeeds right after) but flagging in case it's a symptom of something.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the top-level README and reproduce the two-PC Linux setup using the documented module parameters. Inspect the provider .so matching path alongside /sys/class/infiniband_verbs/uverbs1/device/driver_id and its modalias, then compare the result with ibv_devices and ibv_devinfo. Done means the registered usb4_rdma0 device is claimed by the provider and appears in those userspace tools.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.