kvcache-ai / kvcache-ai/Mooncake

[Store] Make non-conflict standby-restore rejections durable

Open
#4,120 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

[Store] Follow-up: make non-conflict standby-restore rejections durable

Split out from the #3806 review discussion with @Icedcoco.

#3806 makes conflict-derived discards durable: when the tolerant standby restore drops ambiguous overlapping descriptors, it writes a fenced OpLog record per affected key (REMOVE for full drops, PUT_END with only the surviving descriptors for partial ones) and fails the restore unless the batch goes durable.

The other rejection kinds still only produce a log line and a metric: `unknown_endpoint`, `invalid_memory_descriptor`, `capacity_overflow`, `invalid_tenant_id`. A later promotion that replays the same snapshot re-rejects the same entries every round. That is safe and idempotent, but it is repeated work, and for permanently-invalid entries (bad descriptor shape) a durable tombstone would be the cleaner end state.

The subtlety that kept this out of #3806: some rejection reasons are transient. `capacity_overflow` in particular reflects this promotion's capacity view, not the entry's validity, so a durable REMOVE could suppress an entry that a later, roomier promotion should accept. `object_already_exists` / `duplicate_object` must never tombstone either, since the key legitimately exists in the index.

Proposed shape when someone picks this up: a per-reason allowlist on the rejection funnel (`NoteLegacyStandbyRejection`) for the permanently-invalid kinds only, reusing the existing `repair_remove_keys` durable write path, with the transient kinds explicitly excluded.

Not blocking #3806; conflict-derived discards are already durable there.

Contributor guide

Open the contributing guide

Research direction

Start by reading the #3806 review discussion and tracing the rejection funnel at NoteLegacyStandbyRejection, along with the existing repair_remove_keys durable write path. Identify which rejection reasons are permanently invalid versus transient, then make only the allowed reasons durable while explicitly excluding capacity_overflow, object_already_exists, and duplicate_object. Done means later promotion replay no longer repeatedly rejects permanently invalid entries, without suppressing entries that may become valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.