kvcache-ai / kvcache-ai/Mooncake

[RFC]: Self-heal dangling LOCAL_DISK replicas on the read path

Open
#3,884 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Changes proposed

**The gap.** When a COMPLETE LOCAL_DISK replica's backing file physically disappears (manual wipe, RemoveAll broadcast, disk loss), the key stays listed and every read of it fails the same quiet way: the offload RPC read in `real_client.cpp` returns nullptr and Python `get()` returns `b""` with no error. #3711 self-heals exactly this on the Put path (probe file existence, `EvictDiskReplica`, clean retry), but read-only keys never heal by design, so for them the failure is permanent and invisible. This is the still-open half of what #3465 reported.

**Proposal.** On a LOCAL_DISK read failure, probe whether the replica's backing file still exists; if it is gone, evict the dangling replica via `EvictDiskReplica` (idempotent, master-side, same call #3711 already uses), then fail over to another replica if one exists. If none remains, the get fails visibly and the key stops being advertised, instead of returning empty bytes forever.

**Boundaries.** Failure path only, so healthy reads pay nothing. No change to the success path, to replica registration, or to the Python API's success semantics. Multi-replica failover ordering stays as it is today.

**Verification plan.** The in-proc master harness used by #3877's LOCAL_DISK regression test: put, wait for offload, wipe the backing file, then assert the get fails visibly and the key is no longer listed; plus a multi-replica case asserting failover serves the data.

I can implement this if the direction looks right.

Prepared with AI assistance (Kimi K3); the mechanism was verified by reading current main (2b3cefba), including the #3711 Put-path healing boundary and the read failure chain into the Python binding.

Contributor guide

Open the contributing guide

Research direction

Start in real_client.cpp and follow the LOCAL_DISK offload read failure into the Python get() path, then review EvictDiskReplica and the existing Put-path healing from #3711. Use the in-proc master harness from #3877: wipe the backing file after offload, verify a single-replica get fails visibly and the key is no longer listed, and verify multi-replica reads fail over.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.