kvcache-ai / kvcache-ai/Mooncake

[Store] Root fix for the RemoveAll/broadcast wipe-vs-lease divergence

Open
#3,715 6 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

## Summary

Non-forced `RemoveAll` (and the master mailbox broadcast) physically wipes client SSD files without consulting lease state, while the master keeps lease-protected metadata. The divergence this creates (#3709) is currently mitigated on the Put path by #3711, which proves a dangling LOCAL_DISK replica is gone before evicting and retrying. That heal is convergent but lazy: keys that are only ever read, or written through batch paths, stay inconsistent until a Put touches them. The root fix belongs at the wipe boundary.

## Problem

Two independent paths physically delete client SSD files while the master can still hold valid leases:

- `RemoveAll` (non-forced): `storage_backend_->RemoveAll()` on the client wipes every local offload file unconditionally; the master removes only metadata whose leases have expired.
- The master mailbox broadcast tells clients to wipe without carrying per-key lease information.

Once a file is gone but the master's completed replica entry remains, the key is wedged: reads fail and every Put reports `OBJECT_ALREADY_EXISTS`.

## Directions, with their costs

1. **Master returns the exact keys each client may delete.** After lease evaluation the master knows precisely which keys lost protection. Returning that set (or skipping the physical wipe entirely for non-forced RemoveAll) stops the divergence from forming. Cost: an RPC contract change, so old clients against a new master need a defined fallback.
2. **Master enqueues per-client delete sets for offline clients.** Covers clients not online at RemoveAll time. Cost: durable per-client queue state on the master, which also has to replicate in HA mode; that is real new statefulness, not a small patch.
3. **Lease-aware mailbox broadcast.** The broadcast path has no response channel at all, so option 1 alone never covers it; the broadcast would need to carry the lease-checked key set.

The wipe side also races with concurrent Puts in any design (a key re-written between the master's lease check and the client's file deletion), so the Put-time heal in #3711 stays useful as the convergent safety net even after a boundary fix lands.

## Scope of this issue

Track the RemoveAll/broadcast boundary redesign: contract shape, old-client compatibility, whether offline enqueue is worth the durable state, and coverage of the broadcast path. #3711 should be read as mitigation, not resolution.

Prepared with Kimi K3 assistance; the mechanism summary above was re-verified against current main before filing.

Contributor guide

Open the contributing guide

Research direction

Trace the client storage_backend_->RemoveAll() path and the master mailbox broadcast path first, then read #3711 as the existing mitigation. Define the boundary redesign, including old-client compatibility, offline-client handling, and broadcast coverage; done means the agreed design is documented and validated by tests for lease-protected wipes and concurrent Put races.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.