[RFC]: EPD Disaggregation and Cooperative Scheduling for Agentic Multimodal Inference（CCF 赛题四）

### Changes proposed

# RFC: EPD Disaggregation and Cooperative Scheduling for Agentic Multimodal Inference

| Field | Content |
|---|---|
| RFC | RFC-EPD-Agent-Multimodal-2026 |
| Status | Draft |
| Date | 2026-07-05 |
| Target Project | Mooncake / Mooncake Transfer Engine / MooncakeStore |
| Target Integrations | vLLM, SGLang, Qwen-VL/Qwen3-VL-style multimodal models |
| Scope | Encoder-Prefill-Decode disaggregation, Agent state cloning, cross-step KV reuse, evaluation protocol |
| Primary Audience | Mooncake maintainers, serving-system contributors, vLLM/SGLang integration reviewers |
| Implementation Horizon | One-month prototype milestone, followed by staged hardening |
| Compatibility Target | Preserve existing PD and colocated serving paths; new features must be gated by configuration |

---

## Summary

This RFC proposes an extension to Mooncake for **agentic multimodal inference**. The core idea is to generalize Mooncake's KVCache-centric disaggregated serving capability from a two-stage Prefill-Decode model into a broader **Encoder-Prefill-Decode (EPD)** architecture, while also supporting:

- **Agent state cloning** for multi-branch reasoning;
- **visual feature caching** for repeated image/document contexts;
- **cross-step KV reuse** for repeated workflow prefixes;
- **tool-call offload/restore** to release GPU memory during blocking I/O;
- **A2A handoff** for state transfer among planner, executor, verifier, and domain agents;
- **coordinated scheduling** across Encoder, Prefill, Decode, and storage resources.

The proposal introduces a unified `MultimodalState` abstraction that represents visual feature bundles, textual/multimodal KV pages, workflow metadata, state versions, fork branches, TTL, and ownership.

The design intentionally avoids a single global strongly-consistent KV registry. Instead, it uses:

1. **KV Directory** for eventually consistent metadata lookup;
2. **Workflow Shards** for per-workflow strong consistency, refcount, CAS, fork, release, and CoW;
3. **Decentralized Scheduler** for admission control, priority scheduling, offload/restore, and A2A-aware routing.

For KV reuse correctness, the RFC uses a conservative default:

> **Exact Prefix Reuse by default; Approximate Reuse only behind validation gates and feature flags.**

Approximate reuse is never required for correctness. If any validation step fails, the system falls back to exact prefix reuse or full Prefill.

---

## Motivation

### Multimodal inference is naturally EPD-shaped

Multimodal inference contains three computationally different stages:

1. **Encoder**: image/video/audio features are produced by a modality encoder.
2. **Prefill**: the language model processes the full multimodal prompt and builds KV cache.
3. **Decode**: autoregressive generation produces output tokens.

Traditional PD disaggregation couples Encoder with Prefill. This becomes inefficient when the same multimodal context appears repeatedly across turns, agent steps, or branch candidates. Separating Encoder from Prefill enables:

- independent scaling of visual encoder workers;
- reuse of visual features across turns and workflows;
- lower repeated prefill cost for multimodal prompts;
- more flexible routing for prefill-heavy versus decode-heavy workloads.

### Agent workflows require persistent state

Agent workloads are not simple one-shot requests. They involve:

- multi-turn conversations;
- external tool calls and long I/O waits;
- multi-branch reasoning such as Tree-of-Thought, beam search, and verifier branches;
- A2A handoff between planner, executor, retriever, tool agent, and verifier;
- repeated prefixes across workflow steps.

These patterns make KV state a first-class runtime object rather than a temporary per-request artifact.

### Mooncake is well positioned for this extension

Mooncake already focuses on disaggregated KVCache management and high-performance transfer. Extending it to EPD and agentic workflows is a natural next step because the same core primitives are needed:

- cross-node tensor transfer;
- distributed KVCache pooling;
- offload and restore;
- memory ownership;
- scheduling across heterogeneous workers;
- efficient movement of large intermediate states.

This RFC defines a practical architecture that can be implemented incrementally and reviewed in stages.

---

## Goals

This RFC has the following goals.

### G1. Support EPD disaggregation

- Separate Vision Encoder, Prefill, and Decode into independently schedulable workers.
- Transfer Encoder outputs to Prefill and Prefill KV pages to Decode through Mooncake Transfer Engine.
- Preserve fallback paths to normal colocated or PD-only execution.

### G2. Introduce a unified multimodal state abstraction

- Represent FeatureBundle, KVPage, workflow metadata, state versions, fork branches, leases, and TTL.
- Support snapshot-based state reads, page-level refcount, and Copy-on-Write.
- Provide clear lifecycle transitions for active, shared, offloaded, restored, released, and aborted states.

### G3. Support Agent State Cloning

- Enable low-cost fork for multi-branch reasoning.
- Avoid deep-copying full KV state for each branch.
- Make branch writes safe through page-level CoW.
- Make refcount correctness testable and observable.

### G4. Support cross-step KV reuse

- Use exact prefix reuse as the safe default.
- Add approximate or non-contiguous reuse behind validation gates and feature flags.
- Always preserve fallback to full Prefill.

### G5. Reduce control-plane bottlenecks

- Avoid a globally locked GDKR-style metadata service.
- Move refcount and CAS to workflow-local shards.
- Keep cross-workflow metadata eventually consistent with bounded staleness.
- Prevent a single hot workflow or global registry from throttling unrelated workflows.

### G6. Provide a reproducible evaluation protocol

- Define datasets, workload construction, baselines, metrics, stress tests, and passing criteria.
- Evaluate both performance and correctness.
- Require cold-cache and warm-cache results.
- Distinguish throughput improvements from quality regressions.

---

## Non-goals

The following are explicitly out of scope for the initial implementation.

1. **No global linearizability across all KV pages**
- The system does not attempt to make all KV reads/writes globally linearizable.
- Strong consistency is limited to workflow-local metadata and state transitions.

2. **No default approximate reuse without validation**
- Approximate KV reuse is not enabled by default.
- It must be guarded by structural, statistical, and execution validation.

3. **No requirement to support every multimodal architecture in the first version**
- The first implementation targets Qwen-VL/Qwen3-VL-style models or models with similar image-feature injection patterns.
- Other architectures can be integrated later through adapters.

4. **No mandatory RDMA dependency**
- RDMA is an optimization path.
- TCP and shared-memory paths must remain usable for development and basic evaluation.

5. **No production-grade multi-tenant isolation in the first version**
- Basic priority scheduling and fairness metrics are included.
- Full quota management, billing, tenant isolation, and security policy enforcement are left for future RFCs.

6. **No invasive rewrite of vLLM/SGLang internals**
- The design should minimize changes to upstream engines.
- Connector and adapter layers are preferred.

7. **No claim that non-contiguous KV reuse is mathematically equivalent**
- Exact prefix reuse can be treated as safe.
- Non-contiguous reuse is an optimization that must be validated and disabled if quality or correctness gates fail.

---

## Terminology

| Term | Meaning |
|---|---|
| EPD | Encoder-Prefill-Decode disaggregation |
| PD | Prefill-Decode disaggregation |
| FeatureBundle | Encoded visual feature object produced by the Encoder |
| KVPage | Fixed-size page containing KV cache blocks |
| MultimodalState | Unified state object holding feature references, KV page references, workflow metadata, version, epoch, and TTL |
| Workflow | A logical agent task, possibly containing multiple steps, tool calls, forks, and handoff |
| Workflow Shard | Strongly consistent owner of workflow-local metadata |
| KV Directory | Eventually consistent directory for locating states, feature bundles, and KV pages |
| CoW | Copy-on-Write; used when a shared KV page is written by a branch |
| A2A | Agent-to-Agent handoff |
| TTFT | Time to First Token |
| TPOT | Time Per Output Token |
| JCT | Job Completion Time for a workflow |
| SLO | Service-level objective |

---

## Requirements and Invariants

### Functional requirements

1. A multimodal request can run through Encoder, Prefill, and Decode as separate stages.
2. Encoder output can be transferred to Prefill as a `FeatureBundle`.
3. Prefill output can be transferred to Decode as `KVPage` objects.
4. A workflow can fork child states without deep-copying all KV pages.
5. A child branch write must not mutate a parent or sibling branch.
6. Tool-call waiting states can be offloaded and restored.
7. The scheduler can classify workflows into `THINKING`, `INTERACTIVE`, and `HYBRID`.
8. Exact prefix reuse must be available before approximate reuse.
9. Approximate reuse must support fallback.
10. All major operations must export metrics and debug logs.

### Correctness invariants

The following invariants must hold:

1. **Sealed-page immutability**
- A sealed KV page shared by multiple states must never be mutated in place.

2. **Refcount consistency**
- For a given workflow shard, logical page refcount must equal the number of live state references plus in-flight leases.

3. **No use-after-free**
- A page cannot be freed while any live state, transfer, restore, or validation gate holds a lease.

4. **No double-free**
- Page free transitions must be idempotent and recorded.

5. **Snapshot visibility**
- A workflow step reads from one epoch-consistent snapshot.

6. **Fallback safety**
- If reuse validation fails, execution must continue via exact prefix reuse or full Prefill.

7. **A2A source safety**
- During handoff, the source state must not be released until the target either commits or the protocol rolls back.

### Performance requirements

Initial prototype targets should be treated as directional, not contractual:

1. FeatureBundle cache lookup should be significantly cheaper than running Encoder.
2. Fork metadata cost should scale with branch metadata, not full context size.
3. Control-plane operations for unrelated workflows should not block each other.
4. Offload should reduce GPU memory pressure for long tool waits.
5. Warm-cache multi-turn TTFT should improve over cold-cache execution.

---

## Architecture / Design

### High-level architecture

```mermaid
flowchart LR
U[User Client] --> G[Gateway Router]

G --> E[Encoder Workers]
G --> P[Prefill Workers]
E -- FeatureBundle / E-to-P --> P
P -- KV Pages / P-to-D --> D[Decode Workers]

P --> KVD[KV Directory]
D --> WS[Workflow Shards]
WS --> S[Decentralized Scheduler]
S -. load / policy feedback .-> G

E --> Store[MooncakeStore / CPU Cache / NVMe]
P --> Store
D --> Store

WS <--> KVD
S <--> WS
```

The system has four main planes.

### Execution plane

Components:

- Encoder Workers
- Prefill Workers
- Decode Workers

Responsibilities:

- run model computation;
- produce FeatureBundles and KVPages;
- consume restored or transferred state;
- expose stage-level latency and failure metrics.

### Data plane

Components:

- Mooncake Transfer Engine;
- MooncakeStore;
- CPU cache / pinned memory;
- NVMe or object-backed store where applicable.

Responsibilities:

- E-to-P FeatureBundle transfer;
- P-to-D KVPage transfer;
- A2A delta page transfer;
- offload and restore;
- transfer retries, checksums, leases, and idempotency.

### State plane

Components:

- `MultimodalState`;
- `FeatureBundleRef`;
- `KVPageRef`;
- state lifecycle manager;
- local page allocator;
- CoW manager.

Responsibilities:

- represent and mutate workflow state;
- manage state versions and epochs;
- maintain local page ownership and refcount;
- implement fork, release, offload, restore, and CoW.

### Control plane

Components:

- KV Directory;
- Workflow Shards;
- Decentralized Scheduler.

Responsibilities:

- locate state objects and pages;
- maintain workflow-local strong metadata consistency;
- perform scheduling decisions;
- handle failure recovery and ownership transfer.

---

## Detailed Component Design

### Encoder Workers

Encoder workers consume multimodal inputs and produce FeatureBundles.

A FeatureBundle should include:

- `feature_bundle_id`;
- `image_id` or multimodal asset hash;
- model revision;
- feature layer layout;
- dtype and compression;
- shape metadata;
- location hints;
- checksum;
- TTL metadata.

Encoder workers should support:

- warm-cache lookup before encoding;
- async transfer to Prefill;
- CPU/GPU cache registration;
- store-backed persistence if TTL is long.

### Prefill Workers

Prefill workers consume text tokens plus FeatureBundles and generate KVPages.

Prefill workers should support:

- exact prefix lookup;
- FeatureBundle injection into model-specific positions;
- chunked prefill where supported;
- page sealing after generation;
- P-to-D async transfer;
- metrics for prefill tokens/s and page production rate.

A page produced by Prefill should not become shareable until it is sealed.

### Decode Workers

Decode workers consume KVPages and generate tokens.

Decode workers should support:

- on-demand page restore;
- branch fork request;
- CoW on shared-page write;
- cooperative preemption;
- tool-call suspension;
- A2A handoff request;
- decode-local metrics such as TPOT and inter-token latency.

### KV Directory

The KV Directory is intentionally not the source of strong refcount truth.

It stores:

- state location hints;
- page location hints;
- feature bundle location hints;
- version and epoch metadata;
- optional lease metadata;
- last update timestamp.

It must tolerate bounded staleness. Any lookup result must be validated by the consumer before use.

### Workflow Shards

A Workflow Shard owns workflow-local truth.

It stores:

- workflow epoch;
- state version graph;
- page refcount;
- page leases;
- pending transfers;
- pending handoffs;
- local WAL.

Shard-local operations must be idempotent where possible.

### Decentralized Scheduler

The scheduler makes placement decisions using:

- request priority;
- stage service time estimates;
- queue length;
- GPU memory pressure;
- FeatureBundle/KV locality;
- network topology;
- active transfers;
- SLO deadline.

The scheduler is not a global serializing authority. It should make local decisions and use feedback from shards and workers.

---

## Core Design Choice 1: KV Reuse Correctness

This RFC adopts:

> **Exact Prefix Reuse by default, Approximate Reuse with Verify Gate as an optional extension.**

Exact prefix reuse is safe because token prefix, message structure, and position layout match exactly. Approximate reuse is useful for agentic workflows where tool outputs or observations are inserted between otherwise shared context segments. However, non-contiguous reuse can change attention normalization, so it must be guarded.

### Validation pipeline

```mermaid
flowchart LR
A[Reuse Candidate] --> B[Structural Gate]
B -->|pass| C[Statistical Gate]
C -->|pass| D[Execution Gate]
D -->|pass| E[Accept KV Reuse]

B -->|fail| F[Fallback: Full Prefill]
C -->|fail| F
D -->|fail| F
```

### Structural Gate

Checks:

- same model revision;
- same tokenizer revision;
- same LoRA/adaptor configuration;
- same modality injection schema;
- compatible system prompt structure;
- compatible message role order;
- compatible image/document ordering;
- no unsupported middle insertion.

Rejects if semantic order or model configuration differs.

### Statistical Gate

Checks may include:

- token embedding cosine similarity;
- shared-prefix ratio;
- attention entropy drift on a sampled window;
- hidden-state drift where accessible;
- image feature checksum match.

Default policy should be conservative:

- exact prefix reuse is allowed;
- approximate reuse requires high structural compatibility;
- approximate reuse is rejected if shared-prefix ratio is too low.

### Execution Gate

Checks may include:

- run a short validation window;
- compare logits against full recompute for a small number of tokens;
- compare top-k distribution overlap;
- check divergence threshold.

The execution gate is expensive and should be sampled or applied only to risky candidates.

### Reuse modes

| Mode | Default | Safety level | Description |
|---|---:|---|---|
| `none` | no | highest | Always full Prefill |
| `exact_prefix` | yes | high | Reuse only exact token prefix |
| `segment_verified` | no | medium | Reuse compatible segments after validation |
| `aggressive_approx` | no | experimental | More permissive approximate reuse; only for experiments |

---

## Core Design Choice 2: Consistency Model

This RFC chooses:

> **Workflow-level Snapshot Isolation + Bounded-Staleness Directory + Causal Consistency for A2A.**

### Workflow-local consistency

Each workflow owns an epoch. Within an epoch:

- reads see a consistent snapshot;
- forked branches inherit the parent snapshot;
- branch writes create new state versions;
- shared pages are immutable until CoW creates a private copy.

### Cross-workflow metadata

The KV Directory is eventually consistent with bounded staleness. Cross-workflow lookups may see stale locations, but stale entries are validated through page version, epoch, owner, and checksum before use.

### A2A consistency

A2A handoff uses causal consistency plus a two-phase handoff protocol. A target agent must observe the causal predecessor state before continuing execution. Failed handoff must rollback without releasing the source state prematurely.

---

## Core Design Choice 3: Control-plane Disaggregation

Instead of a monolithic GDKR, this RFC defines three components.

### KV Directory

A distributed, eventually consistent directory for metadata lookup.

Responsibilities:

- map `state_id`, `feature_bundle_id`, and `kv_page_id` to candidate locations;
- store version and epoch metadata;
- expose lookup and update APIs;
- tolerate bounded staleness.

The KV Directory does **not** own refcount or CAS.

### Workflow Shards

A workflow shard owns strong metadata consistency for one or more workflows.

Responsibilities:

- maintain workflow epochs and state versions;
- maintain page-level refcount;
- perform fork, release, and CoW transitions;
- validate ownership and page liveness;
- write local WAL for crash recovery.

Refcount and CAS are local to the shard, not global.

### Decentralized Scheduler

The scheduler is distributed across gateway, shard, and worker-side components.

Responsibilities:

- classify tasks into `THINKING`, `INTERACTIVE`, or `HYBRID`;
- perform local admission control;
- estimate stage cost for Encoder, Prefill, and Decode;
- decide offload/restore and prefetch;
- support A2A-aware placement.

---

## State Model

### `MultimodalState`

```python
class MultimodalState:
state_id: str
workflow_id: str
version_id: int
epoch: int
parent_state_id: str | None
parent_version_id: int | None

feature_bundles: list["FeatureBundleRef"]
kv_pages: list["KVPageRef"]

status: str # ACTIVE, OFFLOADING, RESTORING, SHARED, FREE, ABORTED
owner_shard: str
ttl_deadline_ms: int | None
metadata: dict
```

### `FeatureBundleRef`

```python
class FeatureBundleRef:
feature_bundle_id: str
image_id: str
model_revision: str
dtype: str
compression: str
location_hint: list[str]
refcount: int
ttl_deadline_ms: int | None
```

### `KVPageRef`

```python
class KVPageRef:
kv_page_id: str
logical_index: int
page_version: int
owner_worker: str
dtype: str
compression: str
refcount: int
sealed: bool
```

### TransferPolicy

```python
class TransferPolicy:
transport: str # tcp, shm, rdma
compression: str # none, fp8, int8, custom
priority: str # low, normal, high
prefetch: str # none, next_step, speculative
checksum: bool
timeout_ms: int
```

---

## Lifecycle

### State lifecycle

```mermaid
stateDiagram-v2
[*] --> ACTIVE
ACTIVE --> SHARED: fork
ACTIVE --> OFFLOADING: tool wait / memory pressure
OFFLOADING --> RESTORING: resume
RESTORING --> ACTIVE
ACTIVE --> FREE: release
SHARED --> FREE: refcount reaches zero
ACTIVE --> ABORTED: failure
ABORTED --> FREE: cleanup
```

### KV page lifecycle

```mermaid
stateDiagram-v2
[*] --> ALLOCATED
ALLOCATED --> SEALED: prefill complete
SEALED --> REFERENCED: used by state
REFERENCED --> COW_PRIVATE: write on shared page
REFERENCED --> EVICTING: memory pressure
EVICTING --> EVICTED
EVICTED --> RESTORED: on-demand restore
RESTORED --> REFERENCED
REFERENCED --> FREED: refcount zero
```

### Lease model

Transfers, validation gates, and restore operations should hold leases.

A page can be freed only when:

- refcount is zero;
- no active transfer lease exists;
- no active restore lease exists;
- no validation gate lease exists;
- shard-local WAL has recorded the release.

This prevents freeing a page during asynchronous transfer or validation.

---

## Failure Handling

### Worker crash

If an Encoder, Prefill, or Decode worker crashes:

1. Its owner shard marks in-flight states as `SUSPECT`.
2. Pending transfers are retried or cancelled.
3. Pages with committed replicas are restored from Store or peer workers.
4. States without recoverable pages are aborted with a clear error.

### Directory staleness

If KV Directory returns a stale location:

1. The consumer checks page version and owner.
2. If mismatch occurs, the consumer retries directory lookup.
3. If still unresolved, the consumer asks the workflow shard for authoritative state.
4. If no live owner exists, the page is treated as missing and recomputed if possible.

### A2A handoff failure

A2A handoff uses a prepare/commit/abort flow:

1. Source shard prepares handoff and pins source state.
2. Target pulls required pages.
3. Target commits readiness.
4. Source releases or offloads old state.

On timeout or failure, source state remains live and target partial state is cleaned.

### Offload/restore race

If a tool call returns while offload is still in flight:

1. Restore checks the offload transfer state.
2. If the page is still local, restore is converted to a local pin.
3. If transfer already completed, restore pulls from target tier.
4. Duplicate restore calls must be idempotent.

---

## API

The API is intentionally minimal and split into state APIs, transfer APIs, reuse APIs, and scheduler APIs. Actual implementation may map these to C++, Python, gRPC, or internal RPC interfaces.

### State API

#### `create_state`

```python
def create_state(
workflow_id: str,
token_ids: list[int],
feature_bundles: list[FeatureBundleRef],
kv_pages: list[KVPageRef],
ttl_ms: int | None = None,
) -> MultimodalState:
...
```

Creates a new state after Encoder or Prefill. The returned state is owned by a workflow shard.

#### `fork_state`

```python
def fork_state(
parent_state_id: str,
num_branches: int,
reason: str = "agent_branch",
) -> list[MultimodalState]:
...
```

Creates child states that share sealed KV pages and feature bundles with the parent. Refcounts are incremented in the workflow shard.

#### `release_state`

```python
def release_state(
state_id: str,
reason: str = "workflow_complete",
) -> None:
...
```

Releases a state and decrements refcounts. Pages are eligible for GC when refcount reaches zero and all leases expire.

#### `advance_step`

```python
def advance_step(
state_id: str,
new_tokens: list[int],
new_observations: list[str] | None = None,
reuse_policy: str = "exact_prefix",
) -> MultimodalState:
...
```

Advances an agent workflow to a new step and attempts KV reuse according to the reuse policy.

### Transfer API

#### `transfer_feature_bundle`

```python
def transfer_feature_bundle(
feature_bundle_id: str,
src_worker: str,
dst_worker: str,
policy: TransferPolicy,
) -> TransferHandle:
...
```

Transfers visual features from Encoder to Prefill.

#### `transfer_kv_pages`

```python
def transfer_kv_pages(
kv_page_ids: list[str],
src_worker: str,
dst_worker: str,
policy: TransferPolicy,
) -> TransferHandle:
...
```

Transfers KV pages from Prefill to Decode or across agents.

#### `offload_state`

```python
def offload_state(
state_id: str,
target_tier: str, # cpu, nvme, mooncake_store
policy: TransferPolicy,
) -> None:
...
```

Offloads state pages when tool calls or memory pressure make GPU residency inefficient.

#### `restore_state`

```python
def restore_state(
state_id: str,
target_worker: str,
required_pages: list[str] | None = None,
) -> None:
...
```

Restores all or selected pages of a state to a target worker.

### Reuse API

#### `propose_reuse`

```python
def propose_reuse(
state_id: str,
candidate_tokens: list[int],
policy: str,
) -> ReusePlan:
...
```

Builds a reuse plan using exact prefix matching or approximate segment matching.

#### `validate_reuse`

```python
def validate_reuse(
reuse_plan: ReusePlan,
gates: list[str], # structural, statistical, execution
) -> ReuseDecision:
...
```

Runs validation gates and returns accept/fallback decision.

### Scheduler API

#### `submit_workflow`

```python
def submit_workflow(
workflow_trace: dict,
priority_class: str,
slo: dict,
) -> str:
...
```

Submits a workflow and returns a workflow ID.

#### `schedule_next_step`

```python
def schedule_next_step(
workflow_id: str,
state_id: str,
step_metadata: dict,
) -> SchedulingDecision:
...
```

Chooses target workers and transfer/offload strategies for the next workflow step.

### Error semantics

APIs should use explicit error categories:

| Error | Meaning | Recommended handling |
|---|---|---|
| `STALE_DIRECTORY_ENTRY` | Directory location is stale | retry lookup or ask shard |
| `STATE_NOT_FOUND` | State does not exist or was GC'ed | fail request or recompute if possible |
| `PAGE_VERSION_MISMATCH` | Page version does not match expected version | retry or fallback |
| `LEASE_EXPIRED` | Operation lost its lease | reacquire lease or abort |
| `REUSE_REJECTED` | Validation gate rejected reuse | fallback to full Prefill |
| `TRANSFER_TIMEOUT` | Transfer exceeded timeout | retry, change transport, or fallback |
| `A2A_ABORTED` | Handoff aborted | continue on source state if possible |

---

## Configuration

Example configuration:

```yaml
epd:
enabled: true
encoder_pool: "encoder"
prefill_pool: "prefill"
decode_pool: "decode"

reuse:
default_policy: "exact_prefix"
approximate_enabled: false
structural_gate: true
statistical_gate: true
execution_gate: false
max_quality_delta: 0.01

state:
snapshot_isolation: true
ttl_ms: 600000
enable_cow: true
enable_offload: true

directory:
bounded_staleness_ms: 500
backend: "etcd"

scheduler:
queues: ["INTERACTIVE", "HYBRID", "THINKING"]
preemption_enabled: false
admission_control: true

transfer:
default_transport: "tcp"
rdma_enabled: false
checksum: true
compression: "none"
```

---

## Observability

The implementation should export metrics in at least the following groups.

### Stage metrics

- encoder latency;
- E-to-P transfer latency;
- prefill latency;
- P-to-D transfer latency;
- decode TTFT;
- TPOT;
- inter-token latency.

### State metrics

- active states;
- states by status;
- KV pages by status;
- FeatureBundle cache hit rate;
- refcount mismatch count;
- orphan page count;
- lease count;
- GC time.

### Reuse metrics

- exact reuse attempts;
- exact reuse hits;
- approximate reuse attempts;
- gate failure counts by gate type;
- reuse fallback rate;
- quality delta sampling.

### Scheduler metrics

- queue length by priority;
- admission rejection count;
- preemption count;
- victim recovery latency;
- starvation rate;
- SLO attainment.

### Transfer metrics

- bytes transferred by path;
- transfer latency by transport;
- retry count;
- timeout count;
- checksum failure count.

---

## Implementation Plan

The implementation is planned as a one-month engineering sprint. The scope is intentionally staged so that a runnable EPD prototype is available before advanced reuse and A2A features are enabled.

### Week 1: Requirements Clarification and Framework Setup

**Deliverables**

- Finalized interfaces for `MultimodalState`, `KVPageRef`, `FeatureBundleRef`, and `TransferPolicy`.
- Minimal EPD request path scaffold.
- `WorkflowTrace` conversion scripts for initial datasets.
- Initial baseline runners for B0/B1/B4-style comparisons.

**Tasks**

- Review architecture and confirm API boundaries.
- Define owner semantics between Encoder, Prefill, Decode, KV Directory, and Workflow Shards.
- Implement a minimal end-to-end request flow.
- Build small-sample dataset conversion and validation.

**Risks**

- API churn may slow implementation.
- Dataset conversion may introduce evaluation bias.

**Mitigation**

- Freeze MVP interfaces by the end of the week.
- Validate converted traces against original dataset samples.

### Week 2: Basic Feature Implementation

**Deliverables**

- Exact prefix reuse prototype.
- FeatureBundle cache prototype.
- Page-level refcount and CoW fork prototype.
- Basic offload/restore path.
- W0/W1 preliminary evaluation results.

**Tasks**

- Implement RadixTree exact prefix lookup.
- Add FeatureBundle cache lookup and encoder skip path.
- Add workflow-local refcount and CoW page transition.
- Implement offload/restore for tool-wait simulation.
- Run W0 and W1 workloads.

**Risks**

- Refcount mismatch and memory leaks.
- Low prefix hit rate in real workloads.

**Mitigation**

- Add refcount invariants and runtime checks.
- Report hit rate separately from latency benefit.

### Week 3: Advanced Feature Implementation

**Deliverables**

- Approximate reuse behind feature flag.
- Structural/statistical/execution validation gates.
- Initial KV Directory + Workflow Shard split.
- A2A handoff prototype.
- W2/W3/W4 experimental results.

**Tasks**

- Implement segment-level reuse proposal.
- Add validation and fallback logic.
- Move refcount/CAS into workflow-local shard.
- Implement 2PC-style A2A handoff and rollback path.
- Run multi-step, fork, and A2A workloads.

**Risks**

- Approximate reuse may degrade quality.
- A2A may deadlock or leak state.
- Directory/shard split may introduce stale metadata bugs.

**Mitigation**

- Keep approximate reuse disabled by default.
- Add timeout-based rollback and failure injection.
- Validate page version and owner before use.

### Week 4: Integration, Stress Testing, and Reporting

**Deliverables**

- Integrated end-to-end prototype.
- W0-W5 mixed workload evaluation.
- Baseline comparison table.
- Reliability and stress-test report.
- GitHub-ready RFC and implementation summary.

**Tasks**

- Merge feature branches.
- Run B0-B9 comparisons where feasible.
- Run fork, offload/restore, A2A, overload, and reuse stress tests.
- Tune scheduler, TransferPolicy, and cache thresholds.
- Prepare final issue/PR description.

**Risks**

- Integration regressions.
- Insufficient time for deep performance tuning.
- Hardware limitations, especially RDMA availability.

**Mitigation**

- Use feature flags and rollback paths.
- Prioritize correctness before performance.
- Keep TCP/SHM as the default transport path and RDMA as optional.

---

## Current Implementation Status

> The tasks below are **not claimed as complete**. They describe implementation targets and work-in-progress modules. A task is considered complete only after it passes unit tests, end-to-end tests, benchmark validation, and reliability checks.

### Basic Tasks Under Implementation

| Task | Status | Expected validation |
|---|---|---|
| EPD tri-phase separation prototype | In progress | W0 runs end-to-end with measurable E-to-P and P-to-D transfer timing |
| Unified multimodal state object | In progress | all execution stages read/write state through the same API |
| Visual FeatureBundle cache | In progress | W1 reports FeatureBundle hit rate and correct warm-cache behavior |
| RadixTree exact prefix reuse | In progress | exact reuse produces equivalent logits/output under deterministic decoding |
| Page-level CoW fork prototype | In progress | W3 fork tests show stable refcount and no use-after-free |
| Basic offload/restore | In progress | W2 tool-wait workloads complete without state loss |
| Basic scheduler prototype | In progress | scheduler reports per-class queueing and latency metrics |
| Evaluation pipeline | In progress | W0/W1/W2 traces run with fixed seeds and export metrics |

### Advanced Tasks Planned or Partially Implemented

| Task | Status | Notes |
|---|---|---|
| Approximate KV reuse with validation gates | Planned / experimental | feature flag only |
| Three-layer control-plane disaggregation | Planned / partial | start by moving refcount/CAS into workflow-local shards |
| A2A collaboration with 2PC | Planned | requires rollback and failure-injection tests |
| Priority preemption and dynamic routing | Planned | depends on basic scheduler stability |
| Mixed workload and fairness evaluation | Planned | depends on stable core paths |
| Fault injection and recovery | Planned | includes UAF, double-free, WAL, GC validation |
| Tool-call observation handling | Partial / planned | evaluated by W2 workloads |
| Dynamic TransferPolicy | Planned | static TCP/SHM first, adaptive policies later |

---

## Evaluation Plan

### Dataset Construction

Public benchmarks are converted into a unified `WorkflowTrace` format:

```json
{
"workflow_id": "uuid",
"source_dataset": "mnms|gaia|mmmu|mmbench|docvqa|synthetic",
"task_type": "single_turn|multi_turn|tool_use|fork|a2a|offload",
"priority_class": "THINKING|INTERACTIVE|HYBRID",
"images": [
{
"image_id": "content_hash",
"path_or_url": "...",
"reuse_group": "same_image_group_id"
}
],
"steps": [
{
"step_id": 0,
"agent_role": "planner|solver|tool_executor|verifier",
"input_prompt": "...",
"input_token_len": 0,
"shared_prefix_token_len": 0,
"new_token_len": 0,
"expected_tool_call": null,
"expected_answer": "...",
"parent_state_id": null,
"fork_group_id": null
}
],
"gold_answer": "...",
"scoring": {
"type": "exact_match|multiple_choice|f1|anls|llm_judge|tool_success"
}
}
```

### Workloads

| Workload | Purpose | Main capability tested |
|---|---|---|
| W0 single-turn multimodal inference | Validate basic EPD | E/P/D separation |
| W1 multi-turn same-image or multi-image dialog | Validate visual cache | FeatureBundle hit and encoder skip |
| W2 multi-step tool agent | Validate cross-step reuse and offload | JCT, tool wait, step reuse |
| W3 multi-branch fork | Validate Agent State Cloning | CoW, refcount, fork cost |
| W4 A2A handoff | Validate state handoff | 2PC, delta pull, rollback |
| W5 mixed online workload | Validate scheduling and stability | tail latency, fairness, goodput |

### Baselines

| ID | Baseline | Purpose |
|---|---|---|
| B0 | Single-node vLLM/SGLang | Single-node end-to-end baseline |
| B1 | Native PD disaggregation | Compare EPD against PD |
| B2 | PD + text prefix cache | Isolate text-prefix caching |
| B3 | PD + FeatureBundle cache | Isolate visual-feature caching |
| B4 | Naive EPD | Measure separation alone |
| B5 | EPD + deep-copy fork | Compare against CoW fork |
| B6 | EPD + page-level CoW, no cross-step reuse | Isolate cross-step reuse |
| B7 | EPD + exact prefix reuse | Safe reuse baseline |
| B8 | EPD + approximate reuse + validation | Measure aggressive reuse |
| B9 | Full system | Final target system |

### Metrics

Metrics are grouped into seven categories:

1. **Latency**
- TTFT, TPOT, inter-token latency, E2E latency, JCT, step TTFT, handoff latency, offload latency, restore latency, control-plane latency.

2. **Throughput and goodput**
- request/s, workflow/s, output token/s, prefill token/s, effective goodput, SLO attainment, deadline miss ratio, admission success rate, early rejection rate.

3. **Cache and reuse**
- FeatureBundle hit rate, encoder skip rate, prefix token hit rate, KV page hit rate, step reuse rate by token/page/FLOPs, Turn-2+ TTFT reduction, fork memory amplification, CoW rate, reuse fallback rate.

4. **Resources**
- peak HBM, average HBM, KV residency, HBM fragmentation, eviction count, offload bytes, restore bytes, E-to-P bandwidth, P-to-D bandwidth, A2A pull bytes, GPU/NIC utilization, CPU overhead.

5. **Scheduling**
- priority-aware TTFT/JCT, preemption latency, victim recovery latency, starvation rate, queueing delay, routing accuracy, load balance score, Jain fairness index.

6. **Quality**
- accuracy, F1, ANLS, tool-plan success, final-answer correctness, and quality delta versus full recomputation.

7. **Consistency and reliability**
- refcount mismatch count, orphan block count, use-after-free count, double-free count, CoW race failures, 2PC rollback success rate, offload abort success rate, deadlock count, recovery time, data corruption count.

### Passing Criteria

A feature or system variant can be considered successful only if:

1. It improves performance against the relevant baseline.
2. It does not degrade quality beyond the configured threshold.
3. It has zero use-after-free, zero double-free, and zero data corruption.
4. It can be disabled without breaking the baseline path.
5. It exports enough metrics to reproduce and debug the result.

---

## Alternatives Considered

### Alternative A: Global linearizable GDKR

This is simpler to reason about but creates a central bottleneck and conflicts with high-throughput agent workloads. It is rejected for the initial design.

### Alternative B: Fully decentralized per-worker ownership

This removes a central control plane but makes consistency, debugging, and failure recovery significantly harder. It may be revisited later, but the initial design uses workflow shards as an intermediate point.

### Alternative C: Approximate reuse only

This maximizes reuse opportunities but is unsafe without validation and difficult to defend in review. The RFC therefore uses exact prefix reuse as the default and approximate reuse only as an optional validated extension.

### Alternative D: Encoder colocated with Prefill

This is the simplest PD-compatible design but misses the main opportunity for multimodal feature reuse. It remains a fallback path, not the target architecture.

---

## Review Guidance

Reviewers are encouraged to focus on the following questions.

### Architecture review

1. Does the EPD split align with Mooncake's existing transfer and storage abstractions?
2. Are the boundaries between execution plane, data plane, state plane, and control plane clear?
3. Does the design avoid introducing a new global bottleneck?
4. Are workflow-local shards the right unit of consistency and ownership?
5. Are the fallback paths to PD or colocated serving explicit enough?

### Correctness review

1. Is workflow-level snapshot isolation sufficient for agent fork and step advancement?
2. Are stale KV Directory entries safely detected by version, epoch, owner, and checksum validation?
3. Is approximate reuse sufficiently guarded by validation gates?
4. Are fallback paths explicit and safe?
5. Are refcount, lease, and GC invariants testable?

### API review

1. Are the proposed APIs minimal enough for initial integration?
2. Can the APIs be implemented without invasive changes to vLLM or SGLang?
3. Are ownership, lifecycle, and error semantics clear?
4. Should any API be split into lower-level primitives?
5. Are feature flags sufficient to isolate risky functionality?

### Implementation review

1. Is the one-month implementation plan realistic?
2. Which features should be protected behind feature flags?
3. What is the minimal MVP that should be merged first?
4. What tests are required before enabling approximate reuse or A2A handoff?
5. What failure modes must be covered before review approval?

### Evaluation review

1. Are the baselines fair and sufficient?
2. Are the metrics enough to distinguish throughput gains from quality loss?
3. Are stress tests strong enough to expose refcount, CoW, offload, and A2A bugs?
4. Are cold-cache and warm-cache results both required?
5. Should approximate reuse be reported separately from exact prefix reuse?

---

## Open Questions

1. Should approximate reuse be limited to text-only contexts in the first version, or should it support multimodal segments from the beginning?
2. What is the acceptable default threshold for logit divergence in the execution gate?
3. Should the KV Directory be backed by Etcd, TiKV, or a Mooncake-native metadata service?
4. Should workflow shards be colocated with decode workers, or deployed as separate control-plane services?
5. What is the minimal integration path for vLLM and SGLang?
6. What feature flags should be exposed to users versus kept internal?
7. How should memory accounting be represented when FeatureBundle and KV pages are shared across branches?
8. Should exact prefix reuse be implemented first inside the serving engine or as a Mooncake-side adapter?
9. What is the best default TTL for agent states waiting on tool calls?
10. How aggressively should offload be triggered under GPU memory pressure?

---

## Future Work

1. Support additional modalities such as audio and video.
2. Extend the scheduler to multi-tenant quota and fairness policies.
3. Add placement optimization based on network topology and GPU interconnect.
4. Integrate with speculative decoding and verifier-guided branch pruning.
5. Add stronger formal verification for workflow shard refcount and CoW transitions.
6. Support production-grade observability dashboards for state lifecycle and transfer latency.
7. Upstream reusable components to Mooncake, vLLM, and SGLang where appropriate.
8. Explore learning-based scheduling policies after the deterministic scheduler is stable.
9. Add compressed KV formats with quality-aware selection.
10. Add automatic workload classification for agent traces.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start by reviewing Mooncake's existing PD and colocated serving paths, Mooncake Transfer Engine, and MooncakeStore to identify current extension points for Encoder, Prefill, and Decode stages. Compare those entry points with the RFC's MultimodalState, workflow-shard, scheduling, and evaluation requirements; done means a staged, configuration-gated prototype plan with correctness and performance checks.

Written by the indexing model from the issue text.