kvcache-ai / kvcache-ai/Mooncake

Auto GID selection picks a non-routable link-local GID on routed RoCEv2 fabrics using RFC1918 (10.0.0.0/8)

Open
#2,747 2 comments 0 reactions 1 assignee Claimed by @alogfans View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Summary
On a routed RoCEv2 fabric where the RoCE IPv4 addresses live in RFC1918 space
(e.g. 10.0.0.0/8), automatic GID selection (when no explicit GID index is set)
picks a non-routable link-local GID instead of the routable IPv4 GID. Cross-node
QPs then never reach RTR and every cross-node transfer fails.

### Environment
- Transport: RDMA (RoCEv2), auto-GID (no `MC_GID_INDEX` / `gid_index < 0`)
- Fabric: routed RoCEv2, RoCE NIC IPv4 in 10.102.0.0/16 (i.e. inside 10.0.0.0/8)

### GID table (per rdma0, representative)
| index | type | GID | address |
|-------|---------|---------------------------|--------------|
| 0 | RoCE v2 | fe80::0690:81ff:fe37:a9e8 | link-local |
| 1 | RoCE v2 | ::ffff:0a66:0965 | 10.102.9.101 |

### Root cause
`isOverlayIPv4` flags 10.0.0.0/8 (and 172.16/12, 100.64/10) as an "overlay"
address, so the routable IPv4 GID (index 1) is classified as
`kNetworkDegraded`. A network-backed link-local GID (index 0) is also classified
as `kNetworkDegraded`. With both candidates in the same class, the stable
tie-break on ascending `gid_index` selects index 0 — the link-local GID. A
`fe80::` GID cannot route across subnets, so the QP never reaches RTR:

Failed to modify QP to RTR ... timed out [110]

### Impact
Any routed RoCEv2 deployment using RFC1918 10/8 (etc.) addressing must currently
work around this by pinning `MC_GID_INDEX` manually. A flat-L2 fabric that
exposes a normal (non-overlay) routable IPv4 GID is unaffected.

### Proposed fix
Rank a network-backed link-local GID strictly below every network-backed IPv4
GID (routable and degraded/overlay alike), so link-local is only chosen when no
network-backed IPv4 GID exists. PR to follow.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.