kvcache-ai / kvcache-ai/Mooncake
[Store] Root fix for the RemoveAll/broadcast wipe-vs-lease divergence
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
## Summary
Non-forced `RemoveAll` (and the master mailbox broadcast) physically wipes client SSD files without consulting lease state, while the master keeps lease-protected metadata. The divergence this creates (#3709) is currently mitigated on the Put path by #3711, which proves a dangling LOCAL_DISK replica is gone before evicting and retrying. That heal is convergent but lazy: keys that are only ever read, or written through batch paths, stay inconsistent until a Put touches them. The root fix belongs at the wipe boundary.
## Problem
Two independent paths physically delete client SSD files while the master can still hold valid leases:
- `RemoveAll` (non-forced): `storage_backend_->RemoveAll()` on the client wipes every local offload file unconditionally; the master removes only metadata whose leases have expired.
- The master mailbox broadcast tells clients to wipe without carrying per-key lease information.
Once a file is gone but the master's completed replica entry remains, the key is wedged: reads fail and every Put reports `OBJECT_ALREADY_EXISTS`.
## Directions, with their costs
1. **Master returns the exact keys each client may delete.** After lease evaluation the master knows precisely which keys lost protection. Returning that set (or skipping the physical wipe entirely for non-forced RemoveAll) stops the divergence from forming. Cost: an RPC contract change, so old clients against a new master need a defined fallback.
2. **Master enqueues per-client delete sets for offline clients.** Covers clients not online at RemoveAll time. Cost: durable per-client queue state on the master, which also has to replicate in HA mode; that is real new statefulness, not a small patch.
3. **Lease-aware mailbox broadcast.** The broadcast path has no response channel at all, so option 1 alone never covers it; the broadcast would need to carry the lease-checked key set.
The wipe side also races with concurrent Puts in any design (a key re-written between the master's lease check and the client's file deletion), so the Put-time heal in #3711 stays useful as the convergent safety net even after a boundary fix lands.
## Scope of this issue
Track the RemoveAll/broadcast boundary redesign: contract shape, old-client compatibility, whether offline enqueue is worth the durable state, and coverage of the broadcast path. #3711 should be read as mitigation, not resolution.
Prepared with Kimi K3 assistance; the mechanism summary above was re-verified against current main before filing.
Contributor guide
Research direction
Trace the client storage_backend_->RemoveAll() path and the master mailbox broadcast path first, then read #3711 as the existing mitigation. Define the boundary redesign, including old-client compatibility, offline-client handling, and broadcast coverage; done means the agreed design is documented and validated by tests for lease-protected wipes and concurrent Put races.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100