kvcache-ai / kvcache-ai/Mooncake

[TE]: Track RDMA-parity large-buffer support for EFA/CXI/UB

Open
#3,500 1 comment 0 reactions 0 assignees View on GitHub
feature request Transfer Engine
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

## Background

RDMA Transfer Engine already handles buffers larger than the device MR limit end-to-end:

1. Registration auto-chunks into multiple MRs / `BufferDesc`s (#2644).
2. Submit-time slicing caps each WR at the source and target `BufferDesc` remainder (`SliceLengthCalculator` in `RdmaTransport::submitTransferTask`), so a single TE read/write that straddles a chunk seam still works.

Because of that, Mooncake Store no longer needs to split RDMA segments at `max_mr_size`. #3486 is dropping `rdma` from `GetTransportRegistrationLimit()` and leaving the Store-side split in place for EFA, CXI, and UB.

## Goal

Bring EFA, CXI, and UB to the same TE behavior as RDMA, so Store only needs a registration-size split for transports that still cannot register or transfer across chunk seams.

After all three sub-issues land, `GetTransportRegistrationLimit()` should only apply to transports that still lack this support (today that is expected to be none, or UB until its registration shrink is fixed).

## Why this is split into sub-issues

I do not have EFA or CXI hardware to implement and verify the submit-time path, and UB is a different gap (it still shrinks oversized registrations instead of chunking). Tracking them separately keeps owners and test requirements clear.

## Sub-issues

- [x] #3501 EFA: submit-time cross-`BufferDesc` slicing (registration chunking already exists)
- [ ] #3502 CXI: submit-time cross-`BufferDesc` slicing (registration chunking already exists)
- [ ] #3503 UB: registration auto-chunking **and** submit-time cross-`BufferDesc` slicing

## References

- #2644 RDMA registration auto-chunk
- #3486 Store protocol-aware segment split
- `RDMALargeMrTest.WriteStraddlesChunkBoundary` / `WriteWithSourceStraddlingChunkBoundary`

Contributor guide

Open the contributing guide

Research direction

Start with sub-issues #3502 and #3503, then compare the existing RDMA behavior in SliceLengthCalculator and RdmaTransport::submitTransferTask. Review GetTransportRegistrationLimit() and the referenced RDMALargeMrTest cases to identify the corresponding EFA, CXI, and UB coverage. Done means the listed transports support the required registration and cross-BufferDesc transfer behavior so Store-side splitting is no longer needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.