[RFC]: MPComm Transport for TENT — an RDMA-native multi-rail transport backend

### Changes proposed

### Summary

This RFC proposes adding **MPComm** as a TENT transport backend, registered as `TransportType::MPCOMM`.

MPComm (Memory Pool Communication) is an RDMA-native data-plane library for large-scale
heterogeneous memory pools, open-sourced by the Tencent Astral Network Team under Apache-2.0:
. It drives multiple RDMA NICs concurrently with two-level
load balancing (across NICs, and across QPs within a NIC) and NUMA-aware worker placement, exposing
one-sided `put`/`get` primitives.

Data path:

```
TENT Request (WRITE/READ)
-> MpcommTransport::submitTransferTasks()
-> MPComm putAsync() / getAsync() (slicing, NIC/QP selection, workers are internal to MPComm)
-> libmpcomm -> libibverbs
```

Validated scope is **host DRAM and GPU device memory over RoCE/IB**, in both directions and in
mixed combinations. GPU memory is registered through `nvidia-peermem`, the same mechanism
`RdmaTransport` depends on for GPU-Direct. The implementation is complete and validated; this RFC
exists because the change exceeds the 500-LOC threshold in `CONTRIBUTING.md`
("For major architectural changes (>500 LOC excluding tests), we would expect a GitHub issue (RFC)").

Related roadmap: #1058 (Transfer Engine NEXT).

### Motivation

The obvious question is *why a second RDMA transport when `RdmaTransport` already exists*. Three
reasons:

**1. Measured throughput advantage over the existing TENT RDMA path.**
Numbers published in the MPComm repository (2 nodes, single NUMA, 4x CX-7, dual-plane 2x200G bond
per NIC, batch=4, 1GB block):

| Path | Platform | MPComm PUT/GET | Baseline PUT/GET | Ratio |
|---|---|---|---|---|
| Zero-copy | AMD Turin / RTX PRO 5000 | 195.7 / 195.7 GB/s | Mooncake TENT 142.2 / 142.3 | ~1.38x |
| Zero-copy | AMD Genoa / NVIDIA H20 | 165.7 / 152.3 GB/s | Mooncake TENT 122.0 / 111.7 | ~1.36x |
| Non-zero-copy | AMD Turin / RTX PRO 5000 | 91.9 / 84.5 GB/s | UCX 20.9 / 16.7 | up to ~5x |

The gain comes from aggressive multi-rail striping plus per-QP outstanding-request tracking, i.e.
from *scheduling policy inside a purpose-built library*, not from a different verbs API.

**2. It gives TENT a second, independently-developed RDMA implementation.**
Where `RdmaTransport` is Mooncake's own stack, MPComm is maintained and tuned outside the project.
Having both lets deployments pick per workload, and makes it possible to cross-check TENT-level
behaviour against an independent data plane.

**3. Vendor-library integration is an established pattern here.**
`sunrise_link`, `kunpeng_ub` / UB (#3093, #3282), TPU and EFA are all transports that wrap an
external SDK behind the TENT `Transport` interface. MPComm follows the same shape.

### Related work and how this differs

| Transport | Relationship |
|---|---|
| `RdmaTransport` | Same underlying verbs, different scheduling. MPComm is not a replacement — both stay selectable. |
| UB / URMA (#3093, #3282) | Same integration *shape* (new tent-native transport, vendor SDK, `USE_*` gate, enum appended). **Different division of labour** — see the trade-off section below. |
| `sunrise_link` | Closest precedent structurally: TENT-only, vendor SDK, absolute-path library discovery in `common.cmake`. Layout and CMake style were copied from it. |

### Proposed design

#### Architecture

```mermaid
flowchart TB
App["App / Mooncake Store"] --> RT["TENT Runtime
(TransferEngineImpl)"]
RT --> SEL["transport selector
(policy / transport_hint)"]
SEL --> MT["MpcommTransport
(this RFC, ~550 LOC)"]

MT --> API["MPComm API
init / registerMemory / publishBuffer
connect / putAsync / getAsync"]
API --> LIB["libmpcomm"]
LIB --> WK["MPComm internals:
worker threads, slicing,
NIC + QP two-level balancing"]
WK --> V["libibverbs"]

MT -. "publish endpoint" .-> SM["SegmentManager
transport_attrs[MPCOMM]"]
SM -. "peer lookup" .-> MT
```

#### Division of labour, and the trade-off worth discussing

This is the part I most want feedback on, because it **differs from the UB RFC on purpose**.

#3093 states the invariant *"TENT owns scheduling and policy; the URMA adapter only performs native
operations and resource lifecycle."* MPComm cannot follow that invariant: slicing, NIC selection, QP
selection, outstanding-request accounting and worker threads are **inside the library**, and they are
precisely where its throughput advantage comes from. Decomposing them to hand control back to TENT
would mean forking MPComm and giving up the thing we integrate it for.

So the boundary is deliberately coarse:

*What the transport does*
- translate `WRITE`/`READ` requests into `putAsync()` / `getAsync()`, one MPComm transfer per request
- derive MPComm's RDMA device list from the TENT `Topology`
- register/deregister buffers (`registerMemory()` + `publishBuffer()`)
- publish and resolve endpoints through `transport_attrs` (see below)
- poll completions and map MPComm error codes onto TENT task status

*What it deliberately does not do*
- no slicing or spraying (MPComm's job)
- no NIC / QP selection (MPComm's job)
- no separate control-plane message type (reuses `transport_attrs`)
- no re-implementation of MPComm's load balancing inside TENT

**Consequence, stated plainly:** TENT's transport-agnostic facilities that require fine-grained
cooperation are not available for this transport in its first version —
`supportsCancellation()` returns `false`, and there is no bandwidth estimation, no NIC load
statistics, no use of `SubBatch`'s `qp_pool` or progress-notification hooks. Failover and QoS
features depending on those hooks fall back to defaults.

I think this is an acceptable trade-off for a first version, and it is documented rather than
hidden. **Open question O1 below asks whether the maintainers agree**, since it is a precedent
question for wrapping any self-scheduling vendor library.

#### Endpoint bootstrap: reusing `transport_attrs` instead of a new control-plane message

MPComm performs its own TCP metadata handshake and needs the peer's `host:port`. Rather than adding a
UB-style dedicated bootstrap message, the transport publishes `:` into the
local segment's `transport_attrs` under the `MPCOMM` key via
`SegmentManager::updateLocal()` + `synchronizeLocal()`, and peers read it back on first use.

- `updateLocal()` (not writing through the copy-on-write `getLocal()` snapshot) is what makes this
correct.
- Fallback when the attribute is absent: segment name + `MPCOMM_REMOTE_TCP_PORT`.
- Connections are established lazily on first request to a segment and cached per `SegmentID`.

This keeps the metadata plane untouched. Worth confirming this is the intended use of
`transport_attrs` (**open question O2**).

#### Alignment with the TENT `Transport` interface

Implemented: `install` / `uninstall`, `allocateSubBatch` / `freeSubBatch`,
`submitTransferTasks` / `getTransferStatus`, `addMemoryBuffer` / `removeMemoryBuffer`,
`getSupportedTransports` registration.
Not implemented (fall back to base behaviour): cancellation, bandwidth estimation, NIC load stats,
custom allocator (`allocateLocalMemory` is not overridden — the transport is pass-through).

#### File layout

```
mooncake-transfer-engine/tent/
include/tent/transport/mpcomm/mpcomm_transport.h
src/transport/mpcomm/mpcomm_transport.cpp
src/transport/mpcomm/CMakeLists.txt
tests/mpcomm_transport_test.cpp
docs/source/design/transfer-engine/mpcomm_transport.md
```

Registration points touched: `types.h` (enum + name/parse maps), `tent/transfer_engine.h`
(C API macro), `transport_loader.cpp`, `transfer_engine_impl.cpp` (+1 line),
`tent/src/CMakeLists.txt`, `tent/src/transport/CMakeLists.txt`, `pybind.cpp`,
`common.cmake`, benchmark `utils.cpp` / `tent_backend.cpp`, plus docs.

#### Enum and C API stability

`MPCOMM` is **appended at the end** of `TransportType`, immediately after `UB`, keeping existing
serialized values stable — same rule #3282 followed. The C macro `TRANSPORT_MPCOMM` is added with
the matching value. `transportTypeName()` / `parseTransportType()` gain one line each; without them a
`"transports": ["mpcomm"]` policy would silently parse to `UNSPEC`.

#### Build integration

Gated by `USE_MPCOMM=ON`, which requires `MPCOMM_ROOT`. `common.cmake` resolves
`include/mpcomm.h` and `lib/libmpcomm.so` with `find_path` / `find_library` and fails at configure
time with the resolved paths printed — matching the EFA/CXI blocks. The library is linked **by
absolute path**, which also survives `$` when downstream targets (e.g.
`mooncake_master`) link `transfer_engine` privately. With `USE_MPCOMM=OFF` nothing is built,
linked or registered.

### Validation

Completed on 2 nodes with RoCE (multi-NIC, dual NUMA):

| Check | Result |
|---|---|
| Build with `USE_TENT=ON USE_MPCOMM=ON`, no `-Wswitch` warnings | pass |
| Build with `USE_MPCOMM=OFF` | pass |
| `tebench --backend=tent --xport_type=mpcomm`, 2 nodes, DRAM, READ | **> 390 GB/s** aggregate |
| VRAM <-> VRAM (`--seg_type=VRAM` on both sides), i.e. `gpu_to_gpu` | pass, throughput in line with single-GPU expectations |
| VRAM <-> DRAM (VRAM on one side only), i.e. `gpu_to_dram` / `dram_to_gpu` | pass |
| Payload correctness (`--check_consistency=true`) | pass |
| `tent_mpcomm_transport_test` (fork'd two-process WRITE+READ, byte-exact verify) | 2 tests pass |
| `transport_attrs` publish/resolve path exercised | pass — with `MPCOMM_REMOTE_TCP_PORT` deliberately unset, the initiator still reached the peer's advertised port |
| pre-commit (clang-format / cmake-format / codespell) on changed files | pass |

**CI limitation, stated up front.** The functional test needs real RDMA hardware *and*
`libmpcomm.so`, so it self-skips otherwise and cannot validate the data path in CI. UB solved the
equivalent problem with an injectable mock adapter plus a `ub-mock` CI job (#3282). MPComm is
consumed as a shared library through a concrete API rather than an injected interface, so the same
trick needs an interposer or a seam that does not exist today. What CI *can* run today is the
enum/name-mapping test (no hardware). **Open question O3** asks how much more is expected.

### Non-goals

1. Not replacing `RdmaTransport`; both remain selectable.
2. **TENT only** — no legacy Transfer Engine transport path, so `MOONCAKE_PROTOCOL=mpcomm` and
`transfer_engine_bench --protocol=mpcomm` are intentionally unsupported. TENT is where the
project is investing, so a legacy-path backend is not planned.
3. No CXL / file / non-RDMA segment types; MPComm is an RDMA data plane only.
4. No cancellation, QoS or bandwidth-estimation integration in this version.
5. No changes to metadata-plane or control-plane message formats.
6. No re-implementation of MPComm scheduling inside TENT.

### Open questions

- **O1** — Is the coarse-grained delegation acceptable, i.e. a transport that owns its own
scheduling and therefore opts out of cancellation / bandwidth estimation / credit integration? This
sets a precedent for self-scheduling vendor libraries and is the main thing I would like settled
before the PR is reviewed in detail.
- **O2** — Is `transport_attrs` + `updateLocal()` the intended mechanism for transport-specific
endpoint advertisement, or is a dedicated bootstrap message preferred?
- **O3** — What level of CI coverage is expected for a transport whose data path cannot run without
vendor hardware? Is a hardware-gated self-skipping test plus a hardware-free mapping test
sufficient for the first PR?
- **O4** — Naming: keep the functional name `mpcomm` (`USE_MPCOMM`, `tent_xport_mpcomm`), consistent
with `sunrise_link` / `kunpeng_ub`?

### Implementation status

The implementation is finished and validated as tabulated above:

- **~630 LOC of code** (excluding tests and docs) across 14 files; the new transport itself is
~550 LOC, the rest is registration wiring of 1-4 lines per file
- **+259 LOC of tests**, **+342 LOC of docs** (a new `mpcomm_transport.md` plus entries in
`supported-protocols.md`, `build.md` and the design index)
- deliverable as a **single PR**; no phased rollout is needed given the size

I will open the PR once this RFC has been seen, rebased onto current `main` at that point. Feedback
on O1-O3 in particular would shape whether anything needs restructuring first.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start by reading mpcomm_transport.h/.cpp, tests/mpcomm_transport_test.cpp, and the registration points listed in the RFC, then review open questions O1–O4. The implementation is reported complete and validated; done currently means maintainer agreement on the delegation boundary, endpoint advertisement, CI coverage, and naming before review.

Written by the indexing model from the issue text.