[RFC] End-to-End RDMA Data Plane for vLLM-Omni EPD and Omni Stage Transfer

### Changes proposed

# Summary

This RFC proposes connecting the existing Mooncake transfer paths into one end-to-end disaggregated vLLM-Omni pipeline and validating it on a physical multi-node InfiniBand cluster.

The initial target is Qwen3-Omni:

```text
Multimodal Encoder
│ encoder embeddings
▼
Thinker Prefill
│ KV cache
▼
Thinker Decode
│ hidden states / embeddings
▼
Talker
│ codec tensors
▼
Code2Wav
```

The proposal reuses existing infrastructure:

- **Encoder → Prefill:** Mooncake-backed vLLM EC connector
- **Prefill → Decode:** existing vLLM `MooncakeConnector`
- **Thinker → Talker / Talker → Code2Wav:** vLLM-Omni `MooncakeTransferEngineConnector`

The main work is to complete the missing integration, use tensor-native payloads for Qwen3-Omni stage outputs, and provide reproducible validation over real InfiniBand verbs/RDMA rather than only local or simulated deployment.

## Motivation

Mooncake and vLLM-Omni already provide most of the required building blocks, but the complete EPD + Omni pipeline has not yet been integrated and validated as one system.

Related work includes:

- the broader Mooncake and vLLM-Omni collaboration roadmap:
`vllm-project/vllm-omni#2904`
- the Qwen3-Omni tensor-native stage-transfer design:
`vllm-project/vllm-omni#3635`
- the Mooncake-backed vLLM encoder-cache connector:
`vllm-project/vllm#40695`
- the existing Qwen3-Omni Prefill–Decode disaggregation path
- Mooncake roadmap: `#1883`

The remaining gaps are:

1. integrate Encoder → Prefill transfer into the vLLM-Omni pipeline;
2. implement the Qwen3-Omni tensor-native payload contract from
`vllm-project/vllm-omni#3635`;
3. keep large embeddings, hidden states, and codec tensors out of generic Python object serialization;
4. validate the complete pipeline on real multi-node RDMA hardware;
5. identify and upstream any generic Mooncake Transfer Engine issues exposed by the deployment.

Our team has access to a physical InfiniBand cluster and can test actual RDMA transport across machines. The team also includes @xiaohajiayou, an active vLLM-Omni contributor, which allows the model-specific integration to be developed directly against the existing vLLM-Omni execution and configuration paths.

## Proposed Design

```text
Encoder ── embeddings ──► Prefill ── KV cache ──► Decode
Mooncake EC MooncakeConnector

Decode ── hidden states ──► Talker ── codec tensors ──► Code2Wav
MooncakeTransferEngineConnector
```

The ownership boundary remains unchanged:

### vLLM / vLLM-Omni owns

- model-stage scheduling and request lifecycle;
- Qwen3-Omni tensor selection, validation, packing, and reconstruction;
- fallback behavior and model-level correctness;
- destination buffer lifecycle at the execution layer.

### Mooncake owns

- registered CPU/GPU memory transfer;
- TCP and RDMA transport;
- transfer completion and error reporting;
- transport and memory-pool observability.

Large tensors must use tensor-native connector entries. Only compact, versioned metadata should travel through the control path.

A downstream stage must not resume until every required tensor transfer has completed successfully.

## Scope

The initial phase contains three integration tasks.

### 1. Encoder → Thinker Prefill

Use the Mooncake-backed EC connector from `vllm-project/vllm#40695` to run the multimodal Encoder and Thinker Prefill on different workers.

The first implementation may use pinned CPU staging. We will validate that the selected transport is actual InfiniBand verbs/RDMA and separately evaluate a GPUDirect path where the GPU–NIC topology supports it.

### 2. Thinker Prefill → Thinker Decode

Reuse the existing vLLM Mooncake KV-transfer path.

This RFC does not introduce another PD or KV-transfer protocol.

### 3. Qwen3-Omni inter-stage transfer

Implement the non-async tensor-native payload path proposed in
`vllm-project/vllm-omni#3635` for:

- Thinker → Talker
- Talker → Code2Wav

Full-payload transfer is the first milestone. Async codec-chunk streaming remains a follow-up because it requires additional ordering, backpressure, and terminal-event semantics.

## Real RDMA Validation

The integration will be tested on at least two physical machines connected through an InfiniBand switch.

The validation must distinguish:

- TCP;
- IP over InfiniBand;
- InfiniBand verbs/RDMA with pinned host memory;
- GPUDirect RDMA, when supported by the actual PCIe topology.

Published test results will include:

- HCA model and link rate;
- selected Mooncake protocol and RDMA device;
- `ibv_devinfo` / link-state evidence;
- an `ib_write_bw` or equivalent hardware baseline;
- GPU–NIC topology;
- Mooncake runtime logs confirming the selected transport;
- payload sizes, latency, and effective bandwidth.

The end-to-end benchmark will compare the current communication path with Mooncake RDMA for:

- Encoder → Prefill;
- Thinker → Talker;
- Talker → Code2Wav;
- complete request latency and throughput under concurrency.

This validation is intended to complement existing connector work for which public physical multi-node RDMA results are still limited.

## Expected Changes

### vLLM / vLLM-Omni

- connect the Mooncake EC path to the vLLM-Omni Encoder/Prefill deployment;
- add Qwen3-Omni tensor-native packing and reconstruction;
- avoid opaque Python-object transfer for large stage tensors;
- add multi-worker configuration and end-to-end tests.

### Mooncake

The integration should reuse current Transfer Engine APIs where possible.

Mooncake changes should be limited to generic issues discovered during real deployment, such as:

- multi-buffer completion or fencing;
- registered-memory correctness;
- cleanup after timeout or failure;
- transport selection and observability;
- reproducible integration documentation.

Qwen3-Omni-specific payload semantics must not be added to Mooncake core.

## Non-Goals

This RFC does not propose:

- a new scheduler or serving framework;
- replacing the existing Prefill–Decode connector;
- Agent State Cloning;
- Hidden State Prefix Caching;
- approximate KV reuse;
- a new Mooncake metadata service;
- automatic topology-aware placement;
- async codec streaming in the first phase.

## Deliverables

1. Encoder → Prefill transfer over physical InfiniBand RDMA.
2. Reuse of the existing Mooncake PD/KV path.
3. Tensor-native Thinker → Talker full-payload transfer.
4. Tensor-native Talker → Code2Wav full-payload transfer.
5. End-to-end Qwen3-Omni multi-node demo.
6. Reproducible correctness and RDMA benchmark report.
7. Separate upstream PRs for generic Mooncake fixes and vLLM-Omni integration.

## Open Questions

1. Is the current EC connector contract sufficient for Qwen3-Omni Encoder outputs?
2. Should destination buffers be connector-owned or model-runner-owned?
3. Are existing Mooncake completion APIs sufficient for publishing stage-ready metadata safely?
4. Should the first implementation PR cover only Encoder → Prefill, or Encoder → Prefill plus Thinker → Talker?

## Authors

- @GrayMiao123
- @xiaohajiayou

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start by reading the Mooncake-backed vLLM EC connector, the existing vLLM MooncakeConnector, and vLLM-Omni's MooncakeTransferEngineConnector, alongside vLLM-Omni#3635 and vLLM#40695. Validate the design on two physical InfiniBand machines, using ibv_devinfo and ib_write_bw plus Mooncake logs. Done means tensor-native transfers work across all listed stages and reproducible correctness, latency, throughput, and RDMA evidence are published.

Written by the indexing model from the issue text.