[RFC]: Mooncake Store NVMe KV Backend

### Changes proposed

## Summary

This RFC proposes a standards-aligned NVMe KV backend for Mooncake Store.

The design keeps Mooncake’s backend-neutral object interface, adds a dedicated NVMe KV backend and connector/executor stack, separates logical identity from physical key lookup, and keeps device state, placement state, and cleanup orchestration in the control plane.

The backend supports both inline objects and manifest-plus-chunk objects, works across multiple devices, and leaves a clean path for stronger multi-tenant isolation and richer scheduling policies.

## Motivation

Mooncake serves AI storage paths such as KV cache offload, layered storage, and persistence of immutable or append-mostly objects. These workloads need storage that is cheaper and larger than DRAM, more object-native than file or block abstractions, fast enough for latency-sensitive runtime paths, and manageable through Mooncake’s control plane.

NVMe KV is a strong fit because it exposes key-value semantics directly at the device layer. Compared with filesystem-oriented backends, it better matches object-style access, reduces software overhead, enables natural multi-device pooling, and integrates cleanly into Mooncake’s scheduling, observability, and cleanup model.

Representative scenarios include KV cache offload, memory-to-NVMe layered storage, persistent model artifacts, and unified control-plane management of NVMe KV devices.

## Goals / Non-goals

### Goals

- integrate NVMe KV through the same backend abstraction used by other storage backends
- support single-device and multi-device deployment
- define clear backend, connector, and executor responsibilities
- use deterministic physical keys bounded by NVMe KV key-size limits
- preserve correctness under collision, corruption, stale state, and partial write visibility
- support both inline objects and manifest-plus-chunk objects
- keep placement and cleanup authority in the control plane
- distinguish device failure from capacity-full conditions
- support manual cleanup and control-plane-driven automatic fallback cleanup
- reserve identity fields for tenant-, domain-, and namespace-aware isolation

### Non-goals

- hardware-brand-specific backend architecture
- separate backend stacks for KV cache and model weights
- full replica management or global rebalancing in the first version
- complete orphan or foreign-data reconciliation in the initial proposal
- mandatory external console or full OpenAPI surface in phase 1
- mandatory multi-tenant policy engine before backend usability

## Proposal

The proposal adds an NVMe KV backend with four main roles:

### Storage backend

The backend adapts to Mooncake’s shared storage interface and owns object semantics:
- mapping logical objects to physical placement
- choosing object layout
- validating reads
- maintaining a local recovery catalog
- handling runtime decisions for failure, full-state, and retry behavior

### Connector

The connector owns device access semantics:
- resolving configured device selectors
- initializing device access
- exposing device capability metadata
- providing uniform store/retrieve operations by physical key

### Executor

The executor owns low-level transport submission:
- issuing NVMe KV commands
- translating transport/device errors into Mooncake error semantics
- allowing different transport implementations without changing backend semantics

### Control plane

The control plane owns cluster-visible authority:
- device state and schedulability
- object placement state
- pending device-control operations
- cleanup orchestration
- stale-state convergence

The control plane does not proxy the data plane. Object I/O remains in the runtime backend path.

## Data model

### Object identity

The design uses an explicit logical identity abstraction:
- `NvmeKvObjectIdentity`
- `NvmeKvIsolationContext`

The isolation context is structured to support:
- `tenant_id`
- `domain_id`
- `namespace_id`

Even if early deployments leave these empty, the model reserves them now so future multi-tenant isolation does not require redesigning object identity.

### Physical key model

NVMe KV physical keys are bounded by device key-size limits. The design therefore requires:
- arbitrary logical keys at the Mooncake layer
- deterministic mapping from logical identity to bounded physical key
- collision safety so wrong data is never returned

The design uses:
- one deterministic physical key for the root object
- deterministic physical keys for child chunks of large objects
- a separate verify hash stored in object metadata/header

Representative interfaces include:
- `EncodeNvmeKvPhysicalKey(...)`
- `EncodeNvmeKvChunkPhysicalKey(...)`
- `ComputeNvmeKvVerifyHash(...)`

A physical key is a placement key, not proof of identity. Reads must validate header structure, verify hash, payload checksum, and object-type-specific constraints before
returning data.

### Object layout

#### Inline object

Small objects that fit under the effective single-value limit are stored as one object blob:

`[ObjectHeader | optional identity metadata | payload]`

This is the preferred fast path.

#### Manifest-plus-chunk object

Large objects are stored as:
- one root manifest object
- multiple chunk objects

The manifest records enough metadata to reconstruct the full object safely, including chunk placement and integrity information.

### Object header fields

Representative fields include:
- `magic`
- `version`
- `object_type`
- `payload_size`
- `verify_hash`
- `payload_checksum`
- `header_checksum`
- `flags`
- `identity_metadata_size`

Representative identity-related fields include:
- `NvmeKvObjectHeader::kFlagHasIsolationContext`
- `NvmeKvStoredIdentityMetadata`

### Visibility model

Chunked objects must not become visible until the manifest is valid. More generally, the design uses a two-step visibility rule:
1. write device data
2. expose the object only when the committed object is valid

### Metadata and catalog model

The proposal separates two metadata scopes:

- **Local recovery catalog**: persisted by the backend for restart recovery, local lookup, and re-reporting state after restart
- **Control-plane placement authority**: shared authoritative state for device registry, placement records, and pending controls

This split is intentional: backend catalog is a local recovery cache, while the control plane is the shared authority.

## Control plane semantics

The control plane defines cluster-wide semantics for NVMe KV resources.

### Device-state authority

Operator-facing states should include:
- `REGISTERING`
- `HEALTHY`
- `DEGRADED`
- `DRAINING`
- `DISABLED`
- `FAILED`
- `FULL`

### Placement authority

The control plane owns authoritative answers for:
- whether a logical object exists
- which device holds the committed placement
- which physical key is authoritative
- whether the object is committed, deleting, corrupted, or otherwise restricted

This prevents stale on-device data from being treated as live data.

### Shared control core

HTTP and RPC may both be used as entrypoints, but they must share one control core:
- HTTP/RPC adapt protocol
- shared control service owns semantics

### Stale convergence

If a backend stops reporting heartbeat, the control plane must converge stale device state into failed and unschedulable state rather than leaving stale healthy state visible indefinitely.

### Device controls

The control plane may issue explicit operations such as enable or disable. The runtime backend applies them during heartbeat, and subsequent placement follows the updated schedulable set.

### Cleanup authority

Cleanup is control-plane-orchestrated rather than hidden local deletion in the backend. This keeps reclaim observable, auditable, and aligned with placement authority.

## Scheduling and cleanup semantics

### Device scheduling

The initial proposal uses a simple stable policy:
- choose from the enabled-device set
- use hash-based selection over enabled devices
- allow bounded retry across candidates when a write path must move away from a failed target

This is intentionally modest and can evolve later.

### Runtime device states

The backend distinguishes:
- `ENABLED`
- `DISABLED_BY_CONFIG`
- `DISABLED_BY_FAILURE`
- `DISABLED_BY_CAPACITY`

A full device is not the same as a failed device.

### Failure vs capacity-full semantics

Transport and executor layers should map capacity-full conditions to `DEVICE_CAPACITY_FULL` instead of generic write failure.

This preserves the difference between:
- failed device: service/path problem
- full device: healthy but temporarily not writable

That distinction improves scheduling, operator visibility, cleanup behavior, and retry behavior.

### Cleanup modes

The proposal supports two cleanup strategies:

#### Manual cleanup
The caller provides an explicit selector, such as:
- list of keys
- regex selector
- remove-all scope

#### LRU cleanup
The control plane selects reclaim candidates from authoritative placement state, ordered by a recency signal.

### Cleanup safety rules

Cleanup must be conservative and authority-driven. It should not delete objects that are:
- missing from authoritative placement state
- not committed
- still in processing
- lease-protected
- tied to active replication tasks

### Single active cleanup per device

Only one cleanup should run on a device at a time to reduce races and duplicated reclaim work.

### Low-watermark reclaim

Cleanup should reclaim toward a low-watermark target rather than deleting an arbitrary small batch, to avoid repeated full-trigger and tiny-reclaim loops.

### Automatic cleanup fallback

When a batch offload ends in `DEVICE_CAPACITY_FULL`, the runtime may:
1. report fresh backend state to the control plane
2. identify full devices
3. request LRU cleanup for those devices
4. retry the batch once within bounded limits

This keeps recovery control-plane-driven, explicit, bounded, and observable.

### Priority order during full recovery

The cleanup model should remain conservative:

**read > cleanup > new write on the full device**

That means:
- reads should continue for valid committed objects
- cleanup should happen before reusing the full device for new writes
- new writes should prefer other healthy devices when available

## Compatibility

### Runtime compatibility

- upper layers continue using a backend-neutral object interface
- FileStorage remains the runtime hub for offload/load execution
- heartbeat remains the mechanism for applying control-plane decisions and reporting runtime state

### Semantic compatibility

The proposal preserves familiar Mooncake semantics:
- committed objects are returned only when valid
- cleanup is consistent with control-plane authority
- capacity exhaustion is treated differently from path failure

### Forward compatibility

The design leaves room for:
- richer placement strategies
- stronger hotplug handling
- tenant-aware isolation semantics
- asynchronous control-plane task models
- more complete reconcile and orphan-handling flows

## Alternatives considered

### Treat NVMe KV as a special-purpose side path

Rejected because it would leak device-specific logic into upper layers, fragment scheduling behavior, and weaken consistent control-plane integration.

### Collapse connector and backend into one layer

Rejected because it would mix object semantics, placement logic, capability discovery, and transport-specific execution in one component, making the design harder to evolve and less vendor-neutral.

### Use physical key alone as object identity

Rejected because collision or stale device data could cause wrong-object reads. The proposal instead requires verify-hash and checksum validation.

### Let the backend perform hidden local cleanup on full

Rejected because it would bypass control-plane authority, reduce observability, and increase reclaim risk under transient runtime conditions.

### Build a complete reconcile/orphan framework first

Rejected because it would delay delivery of the useful core backend. The first version instead prioritizes correct live-path behavior and correct cleanup of authority-known objects.

## Drawbacks / Risks

- more system complexity than a local-disk backend
- dual metadata surfaces require careful restart and reconciliation behavior
- verify-hash and checksum validation add metadata and read-path cost
- full recovery quality depends on cleanup candidate quality and recency signals
- multi-tenant policy semantics are not complete yet
- some hotplug and shared-device edge cases remain future work

## Open questions

1. Should future placement become capacity-weighted, topology-aware, or replica-aware?
2. What should be the canonical recency signal for LRU cleanup?
3. What is the desired contract for hotplug discovery, disappearance, reappearance, and requalification?
4. What guarantees are needed when multiple hosts access the same NVMe KV device or namespace?
5. What minimal scan/reconcile interface should be standardized after the initial backend lands?
6. When `tenant_id`, `domain_id`, and `namespace_id` become active routing inputs, how should placement, cleanup, and observability be partitioned?
7. Should cleanup, reconcile, drain, and rebuild converge into one asynchronous task framework?
8. What is the community contract for corruption handling, quarantine, and repair after validation failure?

### Before submitting a new issue...

- [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

No implementation files, entry points, or tests are named. Start by reviewing the existing shared storage interface and storage backends, then map the proposed backend, connector, executor, and control-plane responsibilities to the current architecture. Done requires an agreed implementation scope and validation criteria for device access, object layouts, failures, cleanup, and multi-device behavior.

Written by the indexing model from the issue text.