kvcache-ai / kvcache-ai/Mooncake
[Bug][TransferEngine][RDMA] Mixed peer active MTUs are not negotiated, causing RC READ/WRITE failures
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
## Description
Mooncake TransferEngine fails RDMA READ and WRITE when two peers have
different active MTUs, for example 4096 and 1024.
Each endpoint configures `IBV_QP_PATH_MTU` using only its local port:
```cpp
attr.path_mtu = context_.activeMTU();
if (globalConfig().mtu_length < attr.path_mtu)
attr.path_mtu = globalConfig().mtu_length;
```
`HandShakeDesc` does not carry the peer's effective MTU, so the two RC QPs
may be configured with different path MTUs.
## Reproduction
1. Start two TransferEngine peers on different nodes.
2. Peer A RDMA port: `active_mtu=4096`.
3. Peer B RDMA port: `active_mtu=1024`.
4. Run bidirectional RDMA WRITE and READ.
Observed errors:
- `remote invalid request`
- `local access violation`
- `local length error`
- repeated connection re-establishment
Control experiments:
- Native `ib_write_bw` and `ib_read_bw` on the same Pods/VFs pass.
- Mooncake still fails with one QP.
- Mooncake still fails with relaxed ordering disabled.
- Mooncake passes both READ and WRITE when both processes use
`MC_MTU=1024`.
## Expected behavior
The handshake should exchange each endpoint's effective MTU and configure
both sides with:
`negotiated_mtu = min(local_effective_mtu, peer_effective_mtu)`
where:
`local_effective_mtu = min(active_mtu, configured_mtu_limit)`
## Possible fix
- Add an optional effective RDMA MTU field to `HandShakeDesc`.
- Encode and decode it in `TransferHandshakeUtil`.
- Negotiate the common MTU before moving each QP to RTR.
- Apply the same negotiated value to every QP and the two-sided control channel.
- Define backward-compatible behavior when the peer does not provide the field.
- Log local, peer, configured and negotiated MTUs.
Relevant code:
- `mooncake-transfer-engine/include/config.h:61`
- The configured MTU limit defaults to `IBV_MTU_4096`.
- `mooncake-transfer-engine/include/transfer_metadata.h:164-194`
- `HandShakeDesc` does not contain the local effective MTU.
- `mooncake-transfer-engine/src/transfer_metadata.cpp:68-135`
- The handshake encoder/decoder does not exchange MTU information.
- `mooncake-transfer-engine/src/transport/rdma_transport/rdma_endpoint.cpp:1223-1228`
- `path_mtu` is selected only from the local active MTU and configured limit.
- `mooncake-transfer-engine/src/transport/rdma_twosided/ctrl_channel.cpp:281-285`
- The two-sided control channel has the same local-only MTU selection.
## Verified versions
Reproduced on Mooncake commit:
- `3aa4ce2cda2ad3eb09671c6917a3781bed2fd98b`
The same local-only MTU selection is still present on `upstream/main` at:
- `d7da5b92a0298ed531eb48672138a8b9fd563f30`
Relevant source:
- [`HandShakeDesc`](https://github.com/kvcache-ai/Mooncake/blob/d7da5b92a0298ed531eb48672138a8b9fd563f30/mooncake-transfer-engine/include/transfer_metadata.h#L164-L194)
- [`TransferHandshakeUtil`](https://github.com/kvcache-ai/Mooncake/blob/d7da5b92a0298ed531eb48672138a8b9fd563f30/mooncake-transfer-engine/src/transfer_metadata.cpp#L68-L135)
- [`RdmaEndPoint` path MTU selection](https://github.com/kvcache-ai/Mooncake/blob/d7da5b92a0298ed531eb48672138a8b9fd563f30/mooncake-transfer-engine/src/transport/rdma_transport/rdma_endpoint.cpp#L1223-L1228)
- [`CtrlChannel` path MTU selection](https://github.com/kvcache-ai/Mooncake/blob/d7da5b92a0298ed531eb48672138a8b9fd563f30/mooncake-transfer-engine/src/transport/rdma_twosided/ctrl_channel.cpp#L281-L285)
## Result summary
- Default configuration: `write_rc=1`, `read_rc=1`, `passed=false`
- Same Pods/VFs with native `ib_write_bw` and `ib_read_bw`: passed
- Mooncake with `MC_MTU=1024` on both endpoints:
`write_rc=0`, `read_rc=0`, `passed=true`
This issue was drafted with AI assistance and reviewed and verified by the reporter.
Contributor guide
Research direction
Start with HandShakeDesc in mooncake-transfer-engine/include/transfer_metadata.h and the encoder/decoder in src/transfer_metadata.cpp. Then trace path_mtu selection in rdma_endpoint.cpp and ctrl_channel.cpp through QP setup. Done means the handshake supports compatible MTU negotiation, both sides use the negotiated value consistently, and behavior remains defined for peers without the field.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100