[RFC]: Mooncake Data Plane Model Distribution, Phase 1

### Changes proposed

## Summary

This RFC proposes a Phase 1 model distribution capability inside Mooncake's data plane.

In this design:

- Mooncake Store is the storage and transfer substrate for model weights.
- `mooncake_master` exposes HTTP APIs for model distribution orchestration.
- node-side `mooncake_client` executes data movement tasks.
- initial external acquisition is handled by a new `MODEL_DOWNLOAD` task.
- follower replication continues to use Mooncake's existing object copy path.

Phase 1 ends when a model is `Staged` on the requested nodes. It does not include runtime model loading, warmup, or serving readiness.

## Motivation

Mooncake already has the right primitives for model distribution:

- object-based storage
- replica-aware transfer
- task orchestration
- node-side execution

What is missing is a model-aware orchestration path that can do two things in sequence:

1. download model data from an external source onto one seed node and publish it into Mooncake Store
2. fan out the same data to follower nodes through Mooncake's internal replication path

This should be solved inside Mooncake's data plane, not by introducing a new node-level service.

## Goals

Phase 1 must provide the following:

1. `mooncake_master` accepts a model distribution job over HTTP.
2. Mooncake can determine whether a seed node already has the required objects.
3. If not, Mooncake can issue a `MODEL_DOWNLOAD` task to the seed node.
4. The seed node can download model files from a given source and write chunked objects into Mooncake Store.
5. After seed data is available, Mooncake can issue copy tasks to follower nodes.
6. Mooncake can return aggregated job and placement state over HTTP.
7. The terminal success state for Phase 1 is `Staged`.

## Non-Goals

Phase 1 does not attempt to solve:

- runtime model load into serving engines
- runtime warmup
- serving readiness
- scheduler integration
- artifact deletion and GC workflows
- complex multi-source fallback
- cross-cluster distribution

## Scope Boundary

This RFC is intentionally about Mooncake only.

It defines:

- Mooncake's HTTP contract for model distribution
- Mooncake's internal task model
- Mooncake's data movement behavior
- Mooncake's job and placement state model

It does not define the external control plane in detail.

An external control plane, such as `smg`, may call Mooncake's HTTP APIs, track its own higher-level lifecycle state, and present operator-facing workflows. That integration is an example, not the subject of this RFC.

## Terminology

### Model Manifest

An immutable description of one model version, keyed by:

`model_id + revision + format + provider`

The manifest uses file-level semantics, while Mooncake stores the payload as chunked objects.

### Distribution Job

A Mooncake-managed request to stage one model version on a set of target nodes.

### Placement

The per-node staged state of one model version.

### Seed Node

The node responsible for initial external acquisition when the requested objects are not already present.

### Follower Node

A target node that receives the model through Mooncake's internal replication path after seed data becomes available.

## Architecture Overview

```mermaid
flowchart LR
CP[External Control Plane
example: SMG]
MM[mooncake_master
distribution orchestrator]
S[Seed mooncake_client]
F[Follower mooncake_client]
MS[(Mooncake Store objects)]
NV1[(Seed local NVMe)]
NV2[(Follower local NVMe)]

CP -->|Create / Query distribution job| MM
MM -->|MODEL_DOWNLOAD task| S
S -->|publish chunked objects| MS
MM -->|copy tasks| F
MS -->|replicate objects| F
S --> NV1
F --> NV2
```

## Component Responsibilities

### `mooncake_master`

`mooncake_master` is the data plane orchestration entrypoint.

It is responsible for:

- exposing model distribution HTTP APIs
- validating create/query job requests
- checking whether the seed node already has all required objects
- issuing `MODEL_DOWNLOAD` tasks when seed acquisition is needed
- issuing copy tasks to follower nodes after seed data is available
- computing aggregate job and placement state

### node-side `mooncake_client`

The existing Mooncake client process is responsible for execution.

It is responsible for:

- executing `MODEL_DOWNLOAD`
- reading the source file or URL
- writing chunked objects into Mooncake Store
- executing existing copy tasks
- materializing staged model data on local storage

### External Control Plane

An external control plane is out of scope for this RFC, but a typical caller is expected to:

- build or store the model manifest
- select target nodes
- choose the seed node
- create a distribution job in Mooncake
- poll Mooncake for job state

`smg` is one example of such a caller.

## Why There Is No Standalone Model Agent

We explicitly reject a separate node-level model agent for Phase 1.

Reasons:

- it adds one more process and deployment artifact on every node
- it duplicates logic that already belongs in Mooncake's task execution path
- it creates another operational boundary for storage and transfer behavior

The simpler model is:

- `mooncake_master` orchestrates
- `mooncake_client` executes

## End-to-End Workflow

```mermaid
sequenceDiagram
participant CP as External Control Plane
participant MM as mooncake_master
participant Seed as Seed mooncake_client
participant Store as Mooncake Store
participant Follower as Follower mooncake_client

CP->>MM: POST /api/v1/model-distribution/jobs
MM->>MM: Check if seed already has all objects

alt Seed already has objects
MM->>MM: Skip initial download
else Seed missing objects
MM->>Seed: Submit MODEL_DOWNLOAD task
Seed->>Seed: Download source file
Seed->>Store: Publish chunked objects
end

MM->>Follower: Submit copy tasks
Follower->>Store: Pull / materialize objects
CP->>MM: GET /api/v1/model-distribution/jobs?job_id=...
MM-->>CP: Job snapshot with placements
```

### Detailed Flow

1. An external control plane submits a model distribution job to `mooncake_master`.
2. `mooncake_master` checks whether the seed node already has the full required object set.
3. If the seed node does not have the data, `mooncake_master` submits a `MODEL_DOWNLOAD` task.
4. The seed node downloads the model from the given source and writes chunked objects into Mooncake Store.
5. After seed data is available, `mooncake_master` submits copy tasks for follower nodes.
6. Follower nodes stage the model locally using Mooncake's internal replication path.
7. The caller polls `mooncake_master` for job snapshots until the job reaches a terminal state.

## Why a New `MODEL_DOWNLOAD` Task Type

The initial acquisition step is not the same as a copy task.

Copy tasks assume the source object already exists inside Mooncake Store.

Initial model acquisition has different semantics:

- the source may be a local file path or URL
- the seed node must read bytes from outside Mooncake Store
- the seed node must transform file-level content into chunked object-level content

Because of that, a new `MODEL_DOWNLOAD` task type is the correct abstraction for Phase 1.

Follower replication, by contrast, is already object-based and should continue to reuse Mooncake's existing copy task path.

## Data Model

### Manifest Fields

Phase 1 relies on the following file-level fields:

- `path`
- `size_bytes`
- `checksum`
- `download_url`
- `artifact.artifact_key`
- `artifact.object_ids`
- `chunk_layout.chunk_size_bytes`
- `chunk_layout.chunk_count`
- `chunk_layout.object_prefix`

`download_url` is used only for seed-side initial acquisition.

### Distribution Job

The Mooncake distribution job contains:

- `job_id`
- `model_ref`
- `seed_node_id`
- `target_nodes`
- `status`
- `last_error`

### Placement

Each placement contains:

- `node_id`
- `state`
- `local_path`
- `staged_bytes`
- `checksum_verified`
- `last_error`

## API Design

### Mooncake HTTP APIs

Phase 1 introduces the following HTTP endpoints on `mooncake_master`:

- `POST /api/v1/model-distribution/jobs`
- `GET /api/v1/model-distribution/jobs?job_id=...`

Future-compatible but not implemented in Phase 1:

- cancel distribution job
- delete model artifacts

### Example Create Request

```json
{
"job_id": "019d3ca8-c2e7-7d73-ab73-445f9143d1eb",
"model": {
"model_id": "demo-model",
"revision": "r1",
"format": "bin",
"provider": "local"
},
"manifest": {
"model_id": "demo-model",
"revision": "r1",
"format": "bin",
"provider": "local",
"total_size_bytes": 1048576,
"file_manifest_version": "v1",
"files": [
{
"path": "model.bin",
"size_bytes": 1048576,
"checksum": "dummy",
"download_url": "file:///tmp/mooncake-dist-test/model.bin",
"artifact": {
"artifact_key": "artifact-demo-model-r1",
"object_ids": [
"demo-model/r1/chunk-000",
"demo-model/r1/chunk-001",
"demo-model/r1/chunk-002",
"demo-model/r1/chunk-003"
]
},
"chunk_layout": {
"chunk_size_bytes": 262144,
"chunk_count": 4,
"object_prefix": "demo-model/r1"
}
}
]
},
"seed_node_id": "127.0.0.1:25001",
"target_nodes": [
"127.0.0.1:25001",
"127.0.0.1:25002"
],
"replica_policy": {
"target_replica_count": 2
}
}
```

### Example Snapshot Response

```json
{
"job_id": "019d3ca8-c2e7-7d73-ab73-445f9143d1eb",
"status": "replicating",
"placements": [
{
"node_id": "127.0.0.1:25001",
"state": "staged",
"local_path": "/tmp/mooncake-models",
"staged_bytes": 1048576,
"checksum_verified": true,
"last_error": null
},
{
"node_id": "127.0.0.1:25002",
"state": "acquiring",
"local_path": null,
"staged_bytes": 0,
"checksum_verified": false,
"last_error": null
}
],
"last_error": null
}
```

## State Model

### Job States

- `Planned`
- `Seeding`
- `Replicating`
- `Completed`
- `Failed`
- `Canceled`

### Placement States

- `Planned`
- `Acquiring`
- `Staged`
- `Failed`

### State Semantics

- `Staged` is the only success state in Phase 1.
- `Completed` means all requested placements are `Staged`.
- `Seeding` means the seed node is still doing initial acquisition or object publication.
- `Replicating` means seed data is available and follower copy is in progress.
- `Failed` at job level means the overall request did not complete successfully.
- `Failed` at placement level means a specific node did not stage successfully.

```mermaid
stateDiagram-v2
[*] --> Planned
Planned --> Seeding
Planned --> Replicating
Seeding --> Replicating
Seeding --> Failed
Replicating --> Completed
Replicating --> Failed
Planned --> Canceled
Seeding --> Canceled
Replicating --> Canceled

state "Placement" as Placement {
[*] --> PlannedP
PlannedP --> Acquiring
Acquiring --> Staged
Acquiring --> FailedP
PlannedP --> FailedP
}
```

Phase 1 intentionally does not define:

- `Loading`
- `Warming`
- `Serving`

Those belong to later layers above the data plane.

## Scheduling Logic

### Seed Reuse

Before issuing seed download, `mooncake_master` checks whether the seed node already has all required objects.

- if yes, the job skips directly to follower replication
- if no, the job submits `MODEL_DOWNLOAD`

This avoids unnecessary external downloads and enables reuse of previously staged seed data.

### Follower Replication

Follower replication is object-based.

Once seed objects are available:

- `mooncake_master` submits copy tasks per object
- follower nodes acquire and materialize staged data
- placement state is derived from task outcomes

## `MODEL_DOWNLOAD` Task Behavior

The node-side `mooncake_client` executes `MODEL_DOWNLOAD` with the following responsibilities:

1. read the model source from `download_url`
2. create or reuse a local staging path
3. download or read the source bytes
4. split the file according to the manifest chunk layout
5. write each chunk into Mooncake Store using the requested object keys
6. report task completion or failure back to `mooncake_master`

Phase 1 supports simple source forms used for local testing and early rollout:

- `file://...`
- absolute local file path
- HTTP URL, where supported by the current client implementation

## Failure Handling

Phase 1 uses a conservative failure model.

### Seed Failure

If the seed node fails to download or publish the model:

- the distribution job fails
- the seed placement becomes `Failed`
- follower replication does not proceed

### Follower Failure

If one follower fails after seed data is available:

- that follower placement becomes `Failed`
- other placements may still reach `Staged`
- the overall job is considered failed unless all requested placements succeed

### Validation Failure

If local validation fails on a node:

- that placement becomes `Failed`
- `last_error` should contain the reason

## Observability

Mooncake should expose operator-meaningful state at two levels:

### Job-Level

- current distribution job status
- aggregate error state
- high-level progress across placements

### Placement-Level

- per-node state
- staged bytes
- validation result
- last error

The HTTP snapshot returned by `mooncake_master` is the primary integration point for external control planes and operational tooling.

## Resource and Operational Considerations

### Storage

- the seed node needs enough local staging capacity for the downloaded model
- follower nodes need enough local staging capacity for staged replicas
- Mooncake Store object capacity must account for chunked artifacts during replication

### Network

- external bandwidth is consumed only once when seed acquisition is required
- internal bandwidth is consumed for follower replication

### Deployment

This design keeps deployment changes minimal:

- `mooncake_master` gets new distribution HTTP endpoints
- node-side `mooncake_client` gets one new task type

No separate model agent service is required.

## Security Considerations

Phase 1 introduces external file acquisition on the seed node. Before broad production rollout, the following areas must be hardened:

- source allowlists
- URL and path validation
- credential handling for remote sources
- size and checksum enforcement
- staging path isolation

These are follow-up hardening items, not blockers for the Phase 1 architecture.

## Example Integration: External Control Plane

One possible integration pattern is:

1. an external control plane registers or stores the model manifest
2. it selects target nodes and a seed node
3. it calls `POST /api/v1/model-distribution/jobs`
4. it polls `GET /api/v1/model-distribution/jobs?job_id=...`
5. it maps Mooncake placement state into its own higher-level lifecycle state

`smg` is one concrete example of such a control plane, but the Mooncake API should remain generic and not be coupled to `smg`-specific semantics.

## Rollout Plan

Recommended rollout sequence:

1. deploy `mooncake_master` with the new distribution HTTP APIs
2. deploy node-side `mooncake_client` with `MODEL_DOWNLOAD`
3. validate local file source download on one seed node
4. validate follower replication on one additional node
5. validate seed reuse behavior
6. expand to larger artifacts and more follower nodes

## Testing Strategy

Phase 1 should be validated at three levels.

### Unit Tests

- snapshot evaluation logic
- task state to placement state mapping
- manifest to download payload conversion

### Integration Tests

- create distribution job over HTTP
- query distribution snapshot over HTTP
- execute `MODEL_DOWNLOAD` on seed
- execute copy tasks on followers

### End-to-End Validation

The project should maintain a real-interface validation path that covers:

- `mooncake_master` startup
- seed client startup
- follower client startup
- create job over HTTP
- seed download and object publication
- follower replication
- final `Completed` job state and `Staged` placements

## Alternatives Considered

### Reuse `modelexpress`

Rejected.

Reason:

- the goal is to build Mooncake-native model distribution
- this RFC is specifically about replacing that dependency boundary, not extending it

### Add a Standalone Node Agent

Rejected for Phase 1.

Reason:

- adds deployment and configuration burden
- duplicates Mooncake task execution responsibilities

### Put Data Movement in the Control Plane

Rejected.

Reason:

- data movement belongs in Mooncake's data plane
- control planes should orchestrate, not transfer model bytes directly

## Open Questions

1. Should follower failures partially succeed the job or always fail the overall job?
2. How should artifact deletion and storage reclamation be exposed?
3. What is the long-term contract for remote download authentication?
4. Should placement state distinguish "objects exist in store" from "fully materialized on local NVMe" more explicitly?
5. How should later runtime loading phases layer on top of `Staged`?

## Decision

For Phase 1, Mooncake adopts the following approach:

- `mooncake_master` exposes model distribution HTTP APIs
- `mooncake_master` orchestrates seed acquisition and follower replication
- a new `MODEL_DOWNLOAD` task handles initial acquisition
- existing Mooncake copy tasks handle follower replication
- the terminal success state is `Staged`

This is the smallest Mooncake-native design that produces a real, testable, production-shaped model distribution path without adding a new node-level service.

### Before submitting a new issue...

- [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)