vllm-project / vllm-project/aibrix
Engine crashes with ibv_create_ah "UD mlx5" and "RC DEVX QP" no such device
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### 🐛 Describe the bug
2p2d error logs:
- P0 log: ib_device.c:1380 UCX ERROR ibv_create_ah(dlid=49152 sl=0 port=1 src_path_bits=0 dgid=fe80::9e63:c0ff:fe73:bf12 flow_label=0xffffffff sgid_index=0 traffic_class=0) for UD mlx5 connect on mlx5_1 failed: No such device
- P1 log: ib_device.c:1380 UCX ERROR ibv_create_ah(dlid=49152 sl=0 port=1 src_path_bits=0 dgid=fe80::9e63:c0ff:fe72:76e6 flow_label=0xffffffff sgid_index=0 traffic_class=0) for UD mlx5 connect on mlx5_3 failed: No such device
- D0 log: ib_device.c:1380 UCX ERROR ibv_create_ah(dlid=49152 sl=0 port=1 src_path_bits=0 dgid=fe80::9e63:c0ff:fe74:ba44 flow_label=0xffffffff sgid_index=0 traffic_class=0) for UD mlx5 connect on mlx5_6 failed: No such device
- D1 log: ib_device.c:1380 UCX ERROR ibv_create_ah(dlid=49152 sl=0 port=1 src_path_bits=0 dgid=fe80::9e63:c0ff:fe76:303c flow_label=0xffffffff sgid_index=0 traffic_class=0) for RC DEVX QP connect on mlx5_8 failed: No such device
### Steps to Reproduce
1. Deploy sglang 2p2d tp2 for Qwen3-32B.
2. Allocate 2 RDMA devices per pod.
3. Run the pods on the same node.
### Expected behavior
No crash
### Environment
* AIBrix 0.5.0
* VKE
* Node conf: 8 GPUs, 8 RDMA devs
Contributor guide
Research direction
Start with the reported ib_device.c:1380 UCX errors and reproduce the sglang 2p2d Qwen3-32B deployment with two RDMA devices per pod on the same node. Compare the P0, P1, D0, and D1 logs to identify why UD mlx5 and RC DEVX QP connections report "No such device"; done means the deployment runs without a crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ai-infra-agents, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100