kvcache-ai / kvcache-ai/Mooncake
[TE]: Track RDMA-parity large-buffer support for EFA/CXI/UB
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
## Background
RDMA Transfer Engine already handles buffers larger than the device MR limit end-to-end:
1. Registration auto-chunks into multiple MRs / `BufferDesc`s (#2644).
2. Submit-time slicing caps each WR at the source and target `BufferDesc` remainder (`SliceLengthCalculator` in `RdmaTransport::submitTransferTask`), so a single TE read/write that straddles a chunk seam still works.
Because of that, Mooncake Store no longer needs to split RDMA segments at `max_mr_size`. #3486 is dropping `rdma` from `GetTransportRegistrationLimit()` and leaving the Store-side split in place for EFA, CXI, and UB.
## Goal
Bring EFA, CXI, and UB to the same TE behavior as RDMA, so Store only needs a registration-size split for transports that still cannot register or transfer across chunk seams.
After all three sub-issues land, `GetTransportRegistrationLimit()` should only apply to transports that still lack this support (today that is expected to be none, or UB until its registration shrink is fixed).
## Why this is split into sub-issues
I do not have EFA or CXI hardware to implement and verify the submit-time path, and UB is a different gap (it still shrinks oversized registrations instead of chunking). Tracking them separately keeps owners and test requirements clear.
## Sub-issues
- [x] #3501 EFA: submit-time cross-`BufferDesc` slicing (registration chunking already exists)
- [ ] #3502 CXI: submit-time cross-`BufferDesc` slicing (registration chunking already exists)
- [ ] #3503 UB: registration auto-chunking **and** submit-time cross-`BufferDesc` slicing
## References
- #2644 RDMA registration auto-chunk
- #3486 Store protocol-aware segment split
- `RDMALargeMrTest.WriteStraddlesChunkBoundary` / `WriteWithSourceStraddlingChunkBoundary`
Contributor guide
Research direction
Start with sub-issues #3502 and #3503, then compare the existing RDMA behavior in SliceLengthCalculator and RdmaTransport::submitTransferTask. Review GetTransportRegistrationLimit() and the referenced RDMALargeMrTest cases to identify the corresponding EFA, CXI, and UB coverage. Done means the listed transports support the required registration and cross-BufferDesc transfer behavior so Store-side splitting is no longer needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100