[RFC]: Manifest-Driven Heterogeneous KV Cache Resharding and Storage

## Changes proposed

This RFC proposes contracts and planners for moving KV cache between heterogeneous placements. It covers three first-class execution paths:
1. **Runtime-to-Runtime:** Transfer between live GPU/host.
2. **Runtime-to-Store:** upload/offload of KV fragments and publication of a stored manifest.
3. **Store-to-Runtime:** restore/prefetch into a heterogeneous target placement, either directly into final GPU/host pages.

## Motivation
Representative cases include:
- Transfer KVCache to another serving server during rebalance, failover, scale-down, or rolling upgrade;
- GPU KV being offloaded to Mooncake Store, host memory, SSD, or another tier and later restored into a different topology;
- decode-generated KV being reused by a later prefill/extend step in a multi-turn session;
- different topology between P/D node

## Goals and non-goals

### Goals

- Support different source and target TP/PP/DP layouts in one transfer.
- Represent GQA/MQA head replication explicitly rather than inferring it from rank numbers.
- Support any compatible pair of runtime instances, regardless of P/D role or whether the deployment uses P/D disaggregation.
- Support live G2G, Runtime-to-Store, and Store-to-Runtime through consistent resource and fragment semantics.
- Remain independent of SGLang, vLLM, or any model-specific naming rule.
- Allow small, independently reviewable implementation PRs under a complete architecture.

### Non-goals

- Owning request routing, source discovery, replica scheduling, or target traffic activation.
- Owning framework page allocation, page-table mutation, or CUDA tensor construction.
- Repacking between different runtime page sizes
- Inferring model semantics from parameter names, Python classes, or rank conventions inside Mooncake core.
- Requiring NCCL M2N, a particular Store backend, or a particular framework.

## Proposed design
```mermaid
flowchart LR
SD["Snapshot Descriptor
what the KV cache represents"]

subgraph RUNTIME["Runtime-owned
framework owns pin, reserve, and eviction"]
RM["Placement Manifest
parallel placement"]
RB["Runtime Binding
logical ranges → addresses"]
end

subgraph MOONCAKE["Mooncake"]
PLAN["Validation + Logical Planning"]
TE["TE Executor"]
SE["Store Executor"]
end

subgraph STORE["Store-owned
Store owns commit, TTL, and eviction"]
SM["Store Manifest
logical ranges → object ranges"]
OBJ[("Store Objects")]
end

SD --> PLAN
RM --> PLAN
RB --> PLAN
SM -->|"restore source"| PLAN

PLAN -->|"Runtime ↔ Runtime"| TE
PLAN -->|"Runtime → Store"| SE
PLAN -->|"Store → Runtime"| SE

SE -->|"atomically publish after upload"| SM
SM --> OBJ
```
Snapshot Descriptor is the shared content identity. Placement Manifest, Runtime Binding, and Store Manifest are independent input contracts. Runtime and Store own their internal resource lifecycles, while Mooncake only validates, plans, and executes.
- `KVCacheSnapshotDescriptor` describes what the KV cache represents;
- `KVCachePlacementManifest` describes how KV cache is logically distributed by a runtime parallel strategy;
- `KVCacheRuntimeBinding` describes where those logical values reside for one runtime operation;
- `KVCacheStoreManifest` describes how KV cache is organized and persisted inside Store.
### Placement and logical planner

The logical coordinate space is:

```text
(global_layer, K_or_V, logical_token, global_kv_head, head_dim)
```

- **PP:** routing follows explicit global layer ownership in Placement.
- **TP:** planning intersects explicit global KV-head intervals.
- **GQA/MQA replication:** overlapping head ownership is valid only as an exact
declared replica set. Replica ordinal/count are explicit.
- **DP:** one complete source replica is selected for a local target plan.
Runtime-to-Store persists one complete source replica. Target replicas may
independently select a source replica according to an external policy or a
deterministic default.

## Execution paths

### Runtime-to-Runtime

1. The source framework establishes a Snapshot Descriptor and pins the source KV blocks;
2. the target framework allocates and reserves target KV blocks;
3. both sides export Placement Manifests and operation-scoped Runtime Bindings;
4. Mooncake validates snapshot identity, manifest compatibility, logical coverage, and binding bounds;
5. the logical planner computes layer/head intersections and selects one complete source DP replica;
6. the TE executor lowers logical edges into registered-memory operations and executes them;
7. Mooncake collects completion from all expected writers and target participants;
8. the target framework validates the result and activates the target KV cache;
9. both frameworks release pins and reservations.

Mooncake does not determine whether a source block has been evicted. Violating the pin contract between steps 1 and 9 is a framework error.

### Runtime-to-Store

1. The source framework establishes a Snapshot Descriptor and selects one complete source DP replica;
2. it pins the allocation and exports a Placement Manifest and Runtime Binding;
3. Store creates a non-visible upload transaction;
4. Mooncake generates a bounded upload plan and writes logical fragments into Store objects;
5. Store collects receipts and validates object bounds, complete coverage, and checksums;
6. Store atomically publishes an immutable `KVCacheStoreManifest`;
7. the runtime releases its pin after publication; on failure, Store aborts the transaction and exposes no incomplete snapshot.

### Store-to-Runtime

1. Store resolves a published Store Manifest and its Snapshot Descriptor;
2. the target framework validates snapshot semantics and allocates and reserves target KV blocks;
3. the target framework exports a Placement Manifest and Runtime Binding;
4. Mooncake intersects Store logical fragments with the target placement;
5. the Store executor fills target ranges through ranged reads or bounded staging;
6. Mooncake validates read size, coverage, checksums, and completion;
7. the target framework validates the result and activates the KV cache;
8. the framework releases the reservation.

## Compatibility and safety

This protocol is an opt-in, explicit KV reshard API:

- it does not change ordinary Mooncake Store KVCache APIs;
- it does not change existing direct TE operations;
- it does not require migration of existing P/D paths;
- an SGLang adapter only translates its scheduler, allocator, and page table into Runtime contracts;
- Mooncake Core does not depend on SGLang or vLLM Python types or naming conventions.

### Mooncake validates

- strict schemas, field types, integer overflow, duplicate fields, and canonical digests;
- namespace, resource, snapshot, token interval, and semantic fingerprint;
- dtype, item size, K/V dimensions, and layout compatibility;
- Placement Manifest layer/head ownership, replica declaration, and complete DP coverage;
- Logical Plan edge semantics, exact target coverage, and expected writers;
- Runtime Binding references to the Manifest and Snapshot;
- resolved-range logical coverage, address bounds, strides, and registered regions;
- Store Manifest object bounds, logical coverage, checksums, and publication visibility;
- path-specific completion and failure atomicity.

### Mooncake does not validate

- whether a runtime block still belongs to a request;
- whether a runtime evicted or reused a block without Mooncake's knowledge;
- whether a framework-provided model or adapter fingerprint truthfully reflects loaded content;
- framework page tables, allocator reference counts, or scheduler state;
- whether the target runtime should receive external traffic.

## Prototype status and validation

The current prototype establishes the Runtime-to-Runtime contract and one
SGLang P-to-D adapter, but does not yet implement the complete RFC.
Mooncake：https://github.com/LLLL114/Mooncake/tree/feat/mooncake-kv-reshard
Sglang：https://github.com/LLLL114/sglang/tree/feat/mooncake-kv-reshard

Area | Prototype status | Remaining boundary |
| --- | --- | --- |
| KV topology and Placement | Implemented in Mooncake | Stabilize strict wire contracts and snapshot identity |
| Runtime Binding | Buffer-level binding implemented | Define operation-scoped resolved ranges |
| TP/PP/DP planner | Contract and unit tests implemented | Revalidate every deserialized edge semantically and add planning limits |
| GQA/MQA replicas | Explicit replica ordinal/count implemented | Add broader conformance fixtures |
| Runtime-to-Runtime TE | SGLang adapter submits native TE batches | Add multi-node GPU/RDMA E2E and failure recovery validation |
| Runtime-to-Store | Not implemented | Add upload plan, receipts, commit/abort, immutable manifest, integrity |
| Store-to-Runtime | Not implemented | Add stored-source planning and ranged final-page restore

## Alternatives considered

### Framework-only resharding

Frameworks may retain independent rank mappings and page-copy code, but Store
cannot independently restore a snapshot into another Placement and each
framework must reimplement identity, coverage, bounds, and lifecycle checks.

### All-gather before transfer or upload

All-gathering into a canonical TP/PP layout increases peak memory and network
traffic. The proposal preserves source-native fragments and plans direct
logical intersections.

### Persist runtime page IDs

Page IDs are allocation-local and become invalid after restart or restore.
Stored manifests must use logical token ranges and fragment coordinates, not
runtime page-table identities.

### One physical operation type for all three paths

Live TE copy, Store upload, and Store ranged restore have different ownership,
publication, receipt, and staging semantics. They share logical identity and
fragment geometry, but Runtime-to-Store uses a dedicated upload plan and each
executor has its own bounded physical operations.

### Convert layout or dtype in Mooncake core

Conversion requires model- and kernel-specific semantics. The first version is
copy-only and byte-compatible. A future transform executor would require an
explicit transform contract rather than an implicit fallback.

## Phasing

| Phase | Scope |
| --- | --- |
| 1. Core snapshot contract | Stabilize Snapshot Descriptor, Placement Manifest, operation-scoped Runtime Binding, canonical plan identity, semantic plan validation, strict wire contracts, and compatibility adapters. |
| 2. Runtime-to-Runtime executor | Add generic resolved-range validation, TE execution, completion contracts, optional peer warmup, and multi-node content/failure tests; upstream the SGLang adapter and its private allocation lowering as one conformance integration |
| 3. Runtime-to-Store | Add `KVCacheSnapshotUploadPlan`, bounded scatter upload, receipts, checksum, selected-DP coverage, Group integration, commit/abort arbitration, and immutable manifest publication |
| 4. Store-to-Runtime | Add stored-source planning, ranged reads into final pages, DRAM/L3 direct restore, SSD/object-store bounded staging, recovery, and integrity validation |
| 5. Frameworks | Add vLLM and other adapters; publish cross-framework placement, resolved-range, and completion conformance fixtures without moving framework allocation metadata into Mooncake |
| 6. Production lifecycle | Add tenant/retention policy integration, TTL and eviction semantics, observability, placement/plan caching, operation-limit tuning, and scheduler-facing snapshot residency queries

## Related work

- [Manifest-driven heterogeneous model weight resharding and storage RFC
#3111](https://github.com/kvcache-ai/Mooncake/issues/3111)
- [Unified KVCache and Model Weight Management in Mooncake Store RFC
#2282](https://github.com/kvcache-ai/Mooncake/issues/2282)
- Mooncake Transfer Engine P2P handshake, registered-memory, and ranged-transfer APIs
- Current Mooncake `mooncake.reshard.weight` manifest and N-D planner contracts

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start with the RFC's phase table, the Mooncake feat/mooncake-kv-reshard prototype, the SGLang feat/mooncake-kv-reshard adapter, and the existing resharding unit tests mentioned in the status table. This is an architecture spanning three execution paths, so done is not bounded until a phase and implementation boundary are selected and its contracts, validation, tests, and integration criteria are agreed.

Written by the indexing model from the issue text.