kvcache-ai / kvcache-ai/Mooncake
[RFC]: Replica-ID Validation for PutEnd / PutRevoke and GetReplicaList
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Changes proposed
- **Status**: Proposed
- **Module**: Mooncake Store (`mooncake-store`)
## Motivation
### 1. Same-client stale PutEnd / PutRevoke after PutStart preemption
`PutEnd` / `PutRevoke` currently only check that `client_id` matches the object's writer. This rejects a different client after preemption, but fails when the **same client** starts a second write that preempts an earlier in-progress write:
```text
Client C: PutStart#1(key) → replicas R1 (PROCESSING)
… discard timeout / UpsertStart preempts …
Client C: PutStart#2(key) → replicas R2 (PROCESSING); client_id still C
Client C: PutEnd#1 / PutRevoke#1(key) → accepted (client_id matches) → incorrectly completes or revokes R2
```
`PutStart` can reclaim after `put_start_discard_timeout`; `UpsertStart` preempts immediately. A stale `PutEnd` may mark R2 `COMPLETE` before data is written; a stale `PutRevoke` may delete R2 while Put#2 is still in progress.
### 2. False Get failures after lease expiry
On Get, the client first calls `GetReplicaList` (or equivalent) to obtain a replica list and a lease TTL, then transfers data. If the lease expires before the transfer finishes, the client returns `LEASE_EXPIRED` even when the underlying replicas are still valid, to avoid reading data that may have been evicted or replaced. For slow or large transfers this causes avoidable failures: the object may still exist unchanged, but without the replica IDs from `GetReplicaList`, the client cannot ask the master whether those same replicas are still live.
## Proposal
Use **replica IDs** as the durable identity of allocated replicas. `PutStart` / `UpsertStart` / `GetReplicaList` all return them to the client; `PutEnd` / `PutRevoke` and lease-expired Get revalidation consume them.
### Globally unique ReplicaID
`ReplicaID` (`uint64_t`) already exists on each replica. Strengthen allocation as follows:
- On master fresh init, set `next_id_` to a random `uint64`.
- On each new replica, assign `id = next_id_.fetch_add(1)` (atomic).
### Return ReplicaID from PutStart / UpsertStart / GetReplicaList
Every replica descriptor returned by `PutStart`, `UpsertStart`, and `GetReplicaList` (including batch variants) must include a non-zero `ReplicaID`. Clients retain these IDs for the matching write completion or for Get revalidation after lease expiry.
### PutEnd / PutRevoke validate replica IDs
Extend end/revoke RPCs to carry the replica ID from the matching `PutStart` / `UpsertStart`:
```text
PutEnd(client_id, key, tenant_id, replica_type, replica_id)
PutRevoke(client_id, key, tenant_id, replica_type, replica_id)
```
### Lease-expired Get revalidation via GetReplicaList IDs
The client keeps the replica IDs from the initial `GetReplicaList`. After transfer, if the lease has expired, it re-queries the master with those ID. If the same IDs still exist and are `COMPLETE`, return success; otherwise fail as today (`LEASE_EXPIRED` / `INVALID_REPLICA`).
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Contributor guide
Research direction
Start by tracing the Mooncake Store PutStart, UpsertStart, GetReplicaList, PutEnd, and PutRevoke RPC entry points, including their batch variants. Verify how ReplicaID is allocated and returned, then map the client and master paths for lease-expired Get revalidation. Done means stale writes are rejected by ReplicaID and valid expired-lease reads are revalidated against matching COMPLETE replicas.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100