vllm-project / vllm-project/aibrix

Engine crashes with ibv_create_ah "UD mlx5" and "RC DEVX QP" no such device

Open
#1,771 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🐛 Describe the bug

2p2d error logs:
- P0 log: ib_device.c:1380 UCX ERROR ibv_create_ah(dlid=49152 sl=0 port=1 src_path_bits=0 dgid=fe80::9e63:c0ff:fe73:bf12 flow_label=0xffffffff sgid_index=0 traffic_class=0) for UD mlx5 connect on mlx5_1 failed: No such device
- P1 log: ib_device.c:1380 UCX ERROR ibv_create_ah(dlid=49152 sl=0 port=1 src_path_bits=0 dgid=fe80::9e63:c0ff:fe72:76e6 flow_label=0xffffffff sgid_index=0 traffic_class=0) for UD mlx5 connect on mlx5_3 failed: No such device
- D0 log: ib_device.c:1380 UCX ERROR ibv_create_ah(dlid=49152 sl=0 port=1 src_path_bits=0 dgid=fe80::9e63:c0ff:fe74:ba44 flow_label=0xffffffff sgid_index=0 traffic_class=0) for UD mlx5 connect on mlx5_6 failed: No such device
- D1 log: ib_device.c:1380 UCX ERROR ibv_create_ah(dlid=49152 sl=0 port=1 src_path_bits=0 dgid=fe80::9e63:c0ff:fe76:303c flow_label=0xffffffff sgid_index=0 traffic_class=0) for RC DEVX QP connect on mlx5_8 failed: No such device

### Steps to Reproduce

1. Deploy sglang 2p2d tp2 for Qwen3-32B.
2. Allocate 2 RDMA devices per pod.
3. Run the pods on the same node.

### Expected behavior

No crash

### Environment

* AIBrix 0.5.0
* VKE
* Node conf: 8 GPUs, 8 RDMA devs

Contributor guide

Open the contributing guide

Research direction

Start with the reported ib_device.c:1380 UCX errors and reproduce the sglang 2p2d Qwen3-32B deployment with two RDMA devices per pod on the same node. Compare the P0, P1, D0, and D1 logs to identify why UD mlx5 and RC DEVX QP connections report "No such device"; done means the deployment runs without a crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai-infra-agents, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.