kvcache-ai / kvcache-ai/Mooncake
Auto GID selection picks a non-routable link-local GID on routed RoCEv2 fabrics using RFC1918 (10.0.0.0/8)
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Summary
On a routed RoCEv2 fabric where the RoCE IPv4 addresses live in RFC1918 space
(e.g. 10.0.0.0/8), automatic GID selection (when no explicit GID index is set)
picks a non-routable link-local GID instead of the routable IPv4 GID. Cross-node
QPs then never reach RTR and every cross-node transfer fails.
### Environment
- Transport: RDMA (RoCEv2), auto-GID (no `MC_GID_INDEX` / `gid_index < 0`)
- Fabric: routed RoCEv2, RoCE NIC IPv4 in 10.102.0.0/16 (i.e. inside 10.0.0.0/8)
### GID table (per rdma0, representative)
| index | type | GID | address |
|-------|---------|---------------------------|--------------|
| 0 | RoCE v2 | fe80::0690:81ff:fe37:a9e8 | link-local |
| 1 | RoCE v2 | ::ffff:0a66:0965 | 10.102.9.101 |
### Root cause
`isOverlayIPv4` flags 10.0.0.0/8 (and 172.16/12, 100.64/10) as an "overlay"
address, so the routable IPv4 GID (index 1) is classified as
`kNetworkDegraded`. A network-backed link-local GID (index 0) is also classified
as `kNetworkDegraded`. With both candidates in the same class, the stable
tie-break on ascending `gid_index` selects index 0 — the link-local GID. A
`fe80::` GID cannot route across subnets, so the QP never reaches RTR:
Failed to modify QP to RTR ... timed out [110]
### Impact
Any routed RoCEv2 deployment using RFC1918 10/8 (etc.) addressing must currently
work around this by pinning `MC_GID_INDEX` manually. A flat-L2 fabric that
exposes a normal (non-overlay) routable IPv4 GID is unaffected.
### Proposed fix
Rank a network-backed link-local GID strictly below every network-backed IPv4
GID (routable and degraded/overlay alike), so link-local is only chosen when no
network-backed IPv4 GID exists. PR to follow.
Contributor guide
Assessment
This issue has not been assessed yet.