[RoadMap][Call For Contribution] Mooncake Project Overall Roadmap

### Milestone 1: Inference & RL Framework Integration

Deepen integration with mainstream frameworks and enhance support for large-scale architectures.

- [ ] (Agentic-aware): Support agentic-aware KVCache System in inference frameworks.
- [ ] (PD Disaggregation): Optimize PD with more features and support SOTA models.
- [ ] (Hybrid Model): Optimize performance for New Models.
- [ ] (RL Disaggregation): Separate Rollout (sampling) and Update (weight updating) using a globally shared KVCache pool and efficient sample flow for higher PPO/RLHF throughput.

---

### Milestone 2: Store V3 Evolution

Enhance the core storage system for better modularity, elasticity, and high concurrency.

- [ ] (TE & Store Decoupling): Completely separate Transfer Engine (TE) and Store into independent packages.
- [ ] (Tensor-Native & Multi-Replica): Integrate TP rank and model info into APIs; support replica lists for high-frequency KVCache items.
- [ ] (Client/Worker Decoupling): Separate dummy client from real client for flexible, highly available deployment topologies.
- [ ] (Engram-like Backend): Introduce an optimized storage abstraction for N-gram key-value operations to handle highly sparse and concurrent random access.
- [ ] (Object Grouping): Allow specifying multiple objects as part of the same group during put operations, with visibility and eviction handled at the group level
- [ ] (Visibility Enhancement): Further improve visibility by providing more flexible and comprehensive metrics querying capabilities
- [ ] (Memory Allocation Optimization): OffsetAllocator is currently optimized for uniform size distributions, but there are now cases with two or more size classes; targeted optimizations are needed to improve memory utilization

**Store V3 RFCs:**
- Mooncake [#1035](https://github.com/kvcache-ai/Mooncake/issues/1035) — [RoadMap] Mooncake Store V3 Roadmap
- Mooncake [#1836](https://github.com/kvcache-ai/Mooncake/issues/1836) — [RFC] Mooncake Store Multi-Tenant Isolation
- Mooncake [#1732](https://github.com/kvcache-ai/Mooncake/issues/1732) — [RFC] Support Radix Tree Index in Store Master for Prefix Query
- Mooncake [#1772](https://github.com/kvcache-ai/Mooncake/issues/1772) — [RFC] KVPool eviction: evict suffix keys when prefix keys are missing
- Mooncake [#1645](https://github.com/kvcache-ai/Mooncake/issues/1645) — [RFC] Hard Pin and Upsert for Mooncake Store
- Mooncake [#1648](https://github.com/kvcache-ai/Mooncake/issues/1648) — [RFC] Kubernetes Lease-Based HA Without External etcd
- Mooncake [#1650](https://github.com/kvcache-ai/Mooncake/issues/1650) — [RFC] HA OpLog Store Abstraction Layer
- Mooncake [#1673](https://github.com/kvcache-ai/Mooncake/issues/1673) — [RFC] Data Type Support in Mooncake Store
- Mooncake [#1372](https://github.com/kvcache-ai/Mooncake/issues/1372) — [RFC] Support Engram as a storage backend in Mooncake
- Mooncake [#1150](https://github.com/kvcache-ai/Mooncake/issues/1150) — [RFC] Persistence and Recovery Solution for KV Metadata and Segment Information
- Mooncake [#1131](https://github.com/kvcache-ai/Mooncake/issues/1131) — [RFC] Refactor allocation strategy for better performance and flexibility
- Mooncake [#1100](https://github.com/kvcache-ai/Mooncake/issues/1100) — [RFC] Dynamic replica management
- Mooncake [#1062](https://github.com/kvcache-ai/Mooncake/issues/1062) — [RFC] Add Local Cache Mechanism for Mooncake Store Client
- Mooncake [#1054](https://github.com/kvcache-ai/Mooncake/issues/1054) — [RFC] Contribute Local Storage to Distributed Pool
- Mooncake [#972](https://github.com/kvcache-ai/Mooncake/issues/972) — [RFC] More KVCache metrics in both master/client side
- Mooncake [#961](https://github.com/kvcache-ai/Mooncake/issues/961) — [RFC] Fake Client for Mooncake Client
- Mooncake [#1809](https://github.com/kvcache-ai/Mooncake/issues/1809) — [RFC] Integrate Mooncake Store into Dynamo KVCM with Native Rust API *(Closed)*
- Mooncake [#1920](https://github.com/kvcache-ai/Mooncake/issues/1920) — [RFC] Rolling Upgrade Support for Mooncake Store
- Mooncake [#1957](https://github.com/kvcache-ai/Mooncake/issues/1957) — [RFC] Mooncake Store NVMe KV Backend
- Mooncake [#1979](https://github.com/kvcache-ai/Mooncake/issues/1979) — [RFC] Dynamo KVBM + Mooncake Store Integration Design
- Mooncake [#1998](https://github.com/kvcache-ai/Mooncake/issues/1998) — [RFC] Radix Tree Index — Prefix-Aware KV Cache Management and Cascading Eviction
- Mooncake [#2081](https://github.com/kvcache-ai/Mooncake/issues/2081) — [RFC] Simplify Mooncake Store for Maintainability
- Mooncake [#2127](https://github.com/kvcache-ai/Mooncake/issues/2127) — [RFC] Lightweight Best-Effort Group Semantics for Mooncake Store Objects
- Mooncake [#2153](https://github.com/kvcache-ai/Mooncake/issues/2153) — [RFC] Dynamic Tenant Quota and Scoped-Key Isolation

**Store V3 Key PRs (recent):**
- Mooncake [#2080](https://github.com/kvcache-ai/Mooncake/pull/2080) — [Store] Add DSA-like workload allocation strategy *(Merged)*
- Mooncake [#2112](https://github.com/kvcache-ai/Mooncake/pull/2112) — [Store] feat: introduce per-tenant storage quota in master
- Mooncake [#2180](https://github.com/kvcache-ai/Mooncake/pull/2180) — [Store] Add opt-in grouped object routing semantics

---

### Milestone 3: Transfer Engine (TENT) & Networking

Focus on declarative data orchestration and pushing hardware limits.

Release TENT V1.1 at 26Q2 @alogfans

- [ ] (Declarative Orchestration): Implement a plugin-based multi-transport architecture; enable dynamic path selection via hardware auto-discovery and declarative routing policies.
- [ ] (QoS-aware Slice Spraying): Introduce cross-process priority-based slice spraying, minimizing the interference between different tasks; aiming to reduce P99 latency in 8x 400 Gbps network.
- [ ] (Ease of Maintenance): Decouple heterogeneous hardware support from transport protocols via a unified GPU abstraction layer and modular plugin architecture to streamline cross-platform deployment. Improve the self-healing ability.
- [ ] (Transfer Queue): Implement centralized/distributed transfer queue with priority scheduling and backpressure for high-throughput, low-latency data movement under heavy load.
- [ ] (NCCL Support): Use NCCL device API as an optional backend

**Transfer Engine RFCs:**
- Mooncake [#1058](https://github.com/kvcache-ai/Mooncake/issues/1058) — [RoadMap] Mooncake Transfer Engine NEXT
- Mooncake [#1850](https://github.com/kvcache-ai/Mooncake/issues/1850) — [RFC] End-to-End Transfer Tracing for Mooncake
- Mooncake [#1749](https://github.com/kvcache-ai/Mooncake/issues/1749) — [RFC] Optimizing Mooncake TCP Transport for High-Concurrency Throughput
- Mooncake [#1773](https://github.com/kvcache-ai/Mooncake/issues/1773) — [RFC] Enable UB transport via Kunpeng UMDK URMA APIs
- Mooncake [#2117](https://github.com/kvcache-ai/Mooncake/issues/2117) — [RFC] Hierarchical Arch for Intra/Inter Data Center Deployment
- Mooncake [#2123](https://github.com/kvcache-ai/Mooncake/issues/2123) — [RFC] Device Transport — GPU-Initiated Communication for TENT
- Mooncake [#2132](https://github.com/kvcache-ai/Mooncake/issues/2132) — [RFC] TENT Local Transfer Admission Queue
- Mooncake [#2132](https://github.com/kvcache-ai/Mooncake/issues/2630) — [RFC] [RFC]: NCCL device API in Mooncake TENT

**Transfer Engine Key PRs (recent):**
- Mooncake [#2079](https://github.com/kvcache-ai/Mooncake/pull/2079) — [TENT] Add rule-based transport and device selection
- Mooncake [#2090](https://github.com/kvcache-ai/Mooncake/pull/2090) — [TENT] Add cross-platform dynamic library adapter with GPU vendor abstraction
- Mooncake [#2155](https://github.com/kvcache-ai/Mooncake/pull/2155) — [TE] Improving the RDMA transport failure handling
- Mooncake [#2160](https://github.com/kvcache-ai/Mooncake/pull/2160) — [TE] Add explicit progressBatch API

---

### Milestone 4: OS-Level Optimizations

Improve throughput and reduce latency through low-level system integrations.

- [ ] (Decentralized Metadata): Migrate SSD metadata management from clients to a dedicated Master to resolve bottleneck issues.
- [ ] (Multi-Level Resource Pooling): Aggregate VRAM, DRAM, NVMe, and Remote resources into a transparent cache pool using CXL and SHM.

**Related RFCs:**
- Mooncake [#1123](https://github.com/kvcache-ai/Mooncake/issues/1123) — [RFC] Support hugepage memory allocation via mmap to bypass RDMA memory registration limits *(Closed)*
- Mooncake [#2106](https://github.com/kvcache-ai/Mooncake/issues/2106) — [RFC] GPU VRAM (HBM) Pool Support for Mooncake Store

---

### Milestone 5: Routing & Checkpoint Engine

Enhance global scheduling and model weight handling.

- [ ] (KVCache-Aware Routing): Route Decode requests based on underlying KVCache distribution (History Cache) to maximize hit rates and minimize TTFT.
- [ ] (Checkpoint Engine): Strengthen parallel distribution of weights via P2P Store; support native Torch format offload for fast MoE loading.

**Conductor & Routing RFCs:**
- Mooncake [#977](https://github.com/kvcache-ai/Mooncake/issues/977) — [RFC] Mooncake-Conductor: Design and Implementation of a Global Scheduler
- Mooncake [#1767](https://github.com/kvcache-ai/Mooncake/issues/1767) — [RFC] Mooncake Data Plane Model Distribution, Phase 1
- Mooncake [#2189](https://github.com/kvcache-ai/Mooncake/issues/2189) — [Call For Contributions] Mooncake Conductor Roadmap

---

### Milestone 6: Heterogeneous & Cross-Platform Inference (Domestic GPU Support)

Break hardware silos for collaborative deployment. Expand support for domestic accelerator platforms.

- [ ] (Cross-Architecture Collaboration): Enable tasks to flow across different vendors (e.g., Vendor A for high-throughput Prefill, Vendor B for low-latency Decode).
- [ ] (Ascend 910B / Atlas 800T): Adapt Transfer Engine and Store for Huawei Ascend NPU, including Ascend Direct protocol and HCCL integration.
- [ ] (Kunpeng SuperNode UB Transport): Enable UB transport via Kunpeng UMDK URMA APIs for Kunpeng 950 platforms.
- [ ] (Moore Threads MACA/MUSA): Align MACA GPU paths with Mooncake build system; integrate MUSA AI Tensor Engine (MATE) for FA3 attention backend.
- [ ] (MetaX / Yunsilicon): Resolve RDMA compatibility with Yunsilicon XScale NIC and ensure stable deployment.
- [ ] (AMD ROCm / HIP): Add ROCm HIP support to the Mooncake Python package for AMD GPU transport.
- [ ] (AWS EFA): Enhance EFA transport with fi_read, endpoint LRU eviction, and multi-NIC striping.

**Related RFCs:**
- Mooncake [#1773](https://github.com/kvcache-ai/Mooncake/issues/1773) — [RFC] Enable UB transport via Kunpeng UMDK URMA APIs
- Mooncake [#1058](https://github.com/kvcache-ai/Mooncake/issues/1058) — [RoadMap] TENT — Full Heterogeneous Deployment section (Ascend ↔ NVIDIA)
- Mooncake [#1035](https://github.com/kvcache-ai/Mooncake/issues/1035) — [RoadMap] Store V3 — Hardware mentions: AMD, Moore Threads, AWS EFA, Ascend
- Mooncake [#1975](https://github.com/kvcache-ai/Mooncake/issues/1975) — [RFC] Mooncake EP Multi-Vendor GPU Adaptation Design
- Mooncake [#1965](https://github.com/kvcache-ai/Mooncake/issues/1965) — [Feature Request] MACA Support for Mooncake EP/PG

**Tracked issues (Ascend):**
- Mooncake [#1854](https://github.com/kvcache-ai/Mooncake/issues/1854) — [Ascend 910B] rtsHostRegister failed in mooncake-store
- Mooncake [#1107](https://github.com/kvcache-ai/Mooncake/issues/1107) — Mooncake RDMA error on Ascend 910B
- Mooncake [#1766](https://github.com/kvcache-ai/Mooncake/issues/1766) — Failed to modify QP to RTR with Yunsilicon XScale (MetaX)

**Heterogeneous Key PRs (recent):**
- Mooncake [#2118](https://github.com/kvcache-ai/Mooncake/pull/2118) — feat: add Hygon DCU/DTK and Iluvatar CoreX platform support *(Merged)*
- Mooncake [#2196](https://github.com/kvcache-ai/Mooncake/pull/2196) — [Store] enables the Ubtransport for Mooncake Store (Kunpeng)
- Mooncake [#2113](https://github.com/kvcache-ai/Mooncake/pull/2113) — [EFA] Add MC_EFA_CQ_THREADS env var and reduce idle CPU spin

---

### Milestone 7: Unified Resource Management & Multi-Model Serving

Manage KVCache and model weights cohesively on GPUs.

- [ ] (Tiered Weight Storage): Manage weights across VRAM (hot), DRAM (warm), and SSD (cold).
- [ ] (Fine-Grained GPU Sharing): Implement CUDA MPS + custom memory isolation for multi-model sharing.
- [ ] (Dynamic Weight Swapping): Use Mooncake's TE for fast RDMA-based weight migration.
- [ ] (Unified View): Manage both KVCache and weight pools globally.

https://github.com/kvcache-ai/Mooncake/issues/2282

---

### Milestone 8: Fast Instance Startup

Utilize data transfer capabilities to accelerate initialization.

- [ ] (Fast Model Distribution): Pull weights via RDMA from neighboring nodes instead of remote storage.
- [ ] (CUDA Context Restore): Snapshot and restore initialized inference processes to skip warmup.
- [ ] (Warm Pool): Maintain a pool of pre-warmed processes for stateful scaling.
- [ ] (Lazy Pulling): Optimize container images to only pull needed layers.

**Related RFCs:**
- Mooncake [#1767](https://github.com/kvcache-ai/Mooncake/issues/1767) — [RFC] Mooncake Data Plane Model Distribution, Phase 1

---

### Milestone 9: Elasticity & Self-Healing

Leverage decoupled architecture for robust operations.

- [ ] (LLM-Aware Autoscaler): Scale based on global KVCache pressure and Prefix Tree hit rates.
- [ ] (Fast GPU Recovery): Recover failed nodes in <10s by loading weights, utilizing the intact KVCache pool.
- [ ] (sched_ext BPF Scheduler): Ensure lowest-latency CPU scheduling for GPU feeder threads.
- [ ] (TE Fault Tolerance): Automatic fallback to TCP and re-routing on RDMA/node failures: Already supported by TENT.

**Related RFCs:**
- Mooncake [#975](https://github.com/kvcache-ai/Mooncake/issues/975) — [RFC] Cleanup processing replicas if the transferring client is down *(Closed)*
- Mooncake [#1920](https://github.com/kvcache-ai/Mooncake/issues/1920) — [RFC] Rolling Upgrade Support for Mooncake Store

---

### Milestone 10: Agent & Multi-Modal Support

Build the foundation for complex Agent and Multi-Modal workflows.

- [ ] (Agent State Cloning): Fast A2A state sharing and cross-node data transport.
- [ ] (Agent PD Disaggregation): Separate resources for "thinking" vs. "interactive" agents.
- [ ] (EPD Disaggregation): Separate Vision Encoder, Prefill, and Decode for multi-modal scenarios. (Qwen-series supported in sglang)
- [ ] (Diffusion Serving): Explore disaggregated architectures for Diffusion models.

#### Omni Collaboration

- [ ] (Omni Transfer Engine Integration): Integrate Mooncake Transfer Engine as a connector for inter-stage data transfer in the omni multi-modal pipeline (AR → Generation → Diffusion).
- [ ] (Worker-Level Inter-Stage Transfer): Optimize worker-level data movement between omni pipeline stages via high-speed RDMA or shared memory.
- [ ] (Omni Prefix Caching): Support Hidden State Prefix Caching for multi-modal omni models.

**Related RFC**
- https://github.com/vllm-project/vllm-omni/issues/2904
- https://github.com/vllm-project/vllm-omni/issues/3635
- Mooncake [#2098](https://github.com/kvcache-ai/Mooncake/issues/2098) — [RFC] Agent-Aware KV Cache Support in Mooncake (Phase 1)

#### Diffusion Collaboration

- [ ] (Disaggregated Diffusion Deployment): Support decentralized disaggregated deployment for video generation models, decoupling transformer and decoder stages.
- [ ] (Queue-based Orchestration): Implement queue-based scheduling for multi-stage diffusion serving requests.

---

### Milestone 11: KVCache Dynamic Migration

- [ ] (Live Migration): Support lossless, dynamic hot-migration of massive KVCache states across clients/nodes for Spot Instances and elastic scaling.
- [ ] (Migration on Store Instance Exit): Support migrating hot data to other nodes during scale-down or rolling upgrades (optional, enabled manually).
- [ ] (Load Balancing)

**Related RFCs:**
- Mooncake [#1159](https://github.com/kvcache-ai/Mooncake/issues/1159) — [RFC] Replica copy and move support

**Migration Key PRs (recent):**
- Mooncake [#2170](https://github.com/kvcache-ai/Mooncake/pull/2170) — [Store] add copy/move execution api support in master side *(Merged)*

---

### Milestone 12: Hardware Support & ICMS

- [ ] (New Hardware): Support Supernodes, Rubin, PPU, Matrix384.
- [ ] (ICMS): Follow BF4 developments and integrate related capabilities.

---

### Milestone 13: SSD Offload Support

- [x] (SSD offload): Support offloading KV cache to SSD, ensuring higher hit ratio in limited DRAM scenarios.
- [x] (Bandwidth optimization): Optimized bandwidth of SSD loading using io_uring lib.
- [ ] (L2→L1 data promotion): If no memory replica and the object is used, promote it to memory.
- [ ] (Smart eviction strategy): e.g., Selectively evict cold data with LRU-like strategy.

**Related RFCs:**
- Mooncake [#971](https://github.com/kvcache-ai/Mooncake/issues/971) — [RFC] Support distributed read and write operations to SSDs
- Mooncake [#952](https://github.com/kvcache-ai/Mooncake/issues/952) — [RFC] Support Mooncake file eviction in DFS/3FS *(Closed)*
- Mooncake [#1940](https://github.com/kvcache-ai/Mooncake/issues/1940) — [RFC] Implement SSD Pool over NVMe-oF in Mooncake Store
- Mooncake [#1957](https://github.com/kvcache-ai/Mooncake/issues/1957) — [RFC] Mooncake Store NVMe KV Backend
- Mooncake [#1990](https://github.com/kvcache-ai/Mooncake/issues/1990) — [RFC] Native SSD KV Storage Backend for Mooncake Store
- Mooncake [#2025](https://github.com/kvcache-ai/Mooncake/issues/2025) — [RFC] L2→L1 data promotion: If no memory replica and the object is used, promote it to memory
- Mooncake [#2031](https://github.com/kvcache-ai/Mooncake/issues/2031) — [RFC] Promote data from L2 -> L1 when it's not in L1

**SSD Offload Key PRs (recent):**
- Mooncake [#2084](https://github.com/kvcache-ai/Mooncake/pull/2084) — feat(nof): add NVMe-oF SSD cache support
- Mooncake [#2167](https://github.com/kvcache-ai/Mooncake/pull/2167) — [Store] introduce NVMe KV backend core
- Mooncake [#2172](https://github.com/kvcache-ai/Mooncake/pull/2172) — feat(store): add SPDK NoF worker pool
- Mooncake [#2176](https://github.com/kvcache-ai/Mooncake/pull/2176) — [Store] L2->L1 promotion-on-hit: observability metrics + max_per_heartbeat knob

---

### Milestone 14: Mooncake PG

- [ ] **(Transfer Engine): Integrate TENT and refactor transfer engine management.**
- [ ] **(Fault-Tolerance Backend): Improve FT-backend fault-handling and recovery, explore elastic backend design.**
- [ ] (PyTorch c10d API): Lazy initialization & Split group for faster initialization.
- [ ] (Collective Communication): Support ring- and tree-topology algorithm.
- [ ] (Hardware Adaption): Support Ascend, Moore Threads, and AWS EFA.
- [ ] (Reliability): (1) Establish comprehensive unit-testing and benchmark suites (2) Agentic Specification for robust AI-Coding generation.

**Mooncake PG Key PRs & Issues (recent):**
- Mooncake [#2157](https://github.com/kvcache-ai/Mooncake/issues/2157) — [Feature Request] Mooncake PG Fault-Tolerance: Diagnosis & Improvement Plan
- Mooncake [#2149](https://github.com/kvcache-ai/Mooncake/pull/2149) — [PG] support external TE *(Merged)*
- Mooncake [#2182](https://github.com/kvcache-ai/Mooncake/pull/2182) — [PG] Add rank fault tolerance and degraded recovery

---

### Milestone 15: Mooncake EP

- [ ] **(High Throughput Kernel): Implement high throughput kernel for high batch size inference.**
- [ ] (Elastic EP): Support elastically EP extension.
- [ ] (TMA Engine): TMA engine optimization for Hopper-specific architecture.
- [ ] (Benchmark): Comprehensive MoE kernel benchmark under incast traffic.

**Mooncake EP RFCs:**
- Mooncake [#1975](https://github.com/kvcache-ai/Mooncake/issues/1975) — [RFC] Mooncake EP Multi-Vendor GPU Adaptation Design

—

### Milestone 16: Documentation

- [ ] (Update and Refactor Docs): The documentation has become extensive; restructure it and update outdated content
- [ ] (AI-Friendly Knowledge Base): Organize Mooncake’s knowledge into a format that is easy for AI systems to read and consume

------------
To contribute features or provide feedback, please edit this issue directly (if you have write access) or comment below. I will update the roadmap accordingly.

No source file, test, or entry point is named. Start by reviewing the relevant milestone section, its linked RFCs, and recent key PRs, then discuss with maintainers which single roadmap item should be scoped; done requires an agreed, independently actionable contribution with explicit acceptance criteria.

Written by the indexing model from the issue text.