kvcache-ai / kvcache-ai/Mooncake
[Bug] HA standby promotion discards the whole index on one validation failure: "overlapping memory descriptors" -> keys 300 to 0 (v0.3.13)
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
On v0.3.13, promoting an HA standby that **has** the state loses the entire metadata index. The standby had applied every OpLog entry for the data in question, promotion itself succeeded, and then the restore validator rejected the whole `PromotionContext` on a single inconsistency check. The master came up serving with zero keys.
Reproduced **3/3** in one session, at `applied_seq_id` 1920, 2223 and 2526.
### Environment
- `docker.io/kvcacheai/mooncake:0.3.13`, master only (the store clients are the same image)
- Kubernetes, 2 master pods (1 primary + 1 standby), 3 store pods, RDMA
- 3 memory segments x 200 GiB, `replica_num=1`
- Master flags: `-enable_ha=true -ha_backend_type=etcd -ha_backend_connstring=<3-member etcd> -enable_oplog=true -allocation_strategy=free_ratio_first -enable_multi_tenants=true -eviction_high_watermark_ratio=0.90`; `enable_snapshot` and `enable_snapshot_restore` both off
- `-oplog_poll_interval_ms` and `-oplog_batch_max_entries` left at defaults
Note this release already contains #3354 and #3527 — in fact the error string below only exists because of #3354, which is direct evidence that fix is in the running binary.
### Repro
1. Bring up primary + standby with the flags above, wait for the standby to reach `WATCHING`.
2. Let the standby soak under traffic. In our case it had been up **nine minutes before the data was written**, so it watched all 300 puts as they happened.
3. Write 300 x 4 MiB objects. Confirm `master_key_count 300` on the primary and, on the standby, `ha_oplog_applied_sequence_id 1920`, `ha_oplog_applied_entries_total 1953`, `ha_oplog_standby_lag 0`.
4. Delete the primary pod so the standby is promoted.
### What happens
```
I hot_standby_service.cpp:560] Promoting Standby to Primary. Applied seq_id: 1920, lag: 0 entries
I hot_standby_service.cpp:588] Standby promoted to Primary successfully. All remaining OpLog entries have been synced.
E master_service.cpp:3198] RestoreFromStandbySnapshot: overlapping memory descriptors
E master_service_supervisor.cpp:401] Standby restore failed: INVALID_PARAMS
```
`master_key_count` goes **300 -> 0** and stays 0. All three store clients reattach and the full capacity comes back, so the buffers are simply unreachable and will be overwritten. Reads afterwards miss everything; a re-seed + verify returns 300/300, so the cluster is otherwise healthy.
### Two separate problems
**1. The validator is all-or-nothing, and the failure is unobservable.**
`master_service_supervisor.cpp` calls `RestoreFromStandby` and, on failure, only logs:
```cpp
if (!restore_result) {
LOG(ERROR) << "Standby restore failed: " << toString(restore_result.error());
}
```
Execution continues and the new primary serves an empty index. `MasterService::RestoreFromStandbySnapshot` (`master_service.cpp:2990`) has roughly ten `return tl::make_unexpected(...)` branches — unknown endpoint, invalid descriptor, capacity overflow, duplicate object, overlapping ranges — and **any one of them discards every object in the context**, including the ones that validated cleanly.
For a KV cache this trade seems backwards: dropping the offending replica (or the offending segment's replicas) and restoring the rest would preserve almost all of the index, and prefix reuse degrades gracefully. Losing 100% of it to protect against one bad descriptor does not.
There is also **no metric** for this. `ha_oplog_standby_lag` read `0` throughout, which is honest — it means "not behind on the stream I am watching" and says nothing about whether the context will pass validation. Those two `E` lines are the only evidence that anything went wrong, so an operator watching dashboards sees a clean failover and a mysteriously cold cache. A counter such as `ha_standby_restore_failures_total` (ideally labelled by `ErrorCode`), plus the number of objects accepted vs rejected, would make this visible.
**2. Why is the overlap check firing at all?**
The check at `master_service.cpp:3193-3202` sorts the `(buffer_address_, size_)` pairs per segment and rejects if any two intervals overlap. `memory_ranges` is populated at `master_service.cpp:3139`, gated on `desc.status != REMOVED && desc.status != FAILED`.
Our hypothesis — offered as a hypothesis, we have not instrumented it — is that the standby's replayed view legitimately contains two live-looking replicas over the same address: the primary freed a buffer and reallocated it, and the standby's OpLog state at the promotion instant still carries the old replica in a non-`REMOVED`/`FAILED` status. Under a vLLM KV workload with eviction running (`eviction_high_watermark_ratio=0.90`), address reuse is constant, so this would be expected rather than exceptional.
If that is right, the invariant the check assumes does not hold on a live cache, and the check — not the state — is the bug. If it is wrong, we would appreciate knowing what is supposed to guarantee non-overlap, and we are happy to run instrumented builds to dump the offending pair.
### What we would like
1. Make the restore tolerant: skip the objects that fail validation and restore the rest, instead of returning `INVALID_PARAMS` for the whole context.
2. Log the offending descriptors (segment, addresses, sizes, both statuses) so this is diagnosable without a custom build.
3. Add a metric for standby restore failure and for objects accepted/rejected.
4. Clarify whether overlapping ranges are expected under eviction and address reuse.
### Notes
`ha_oplog_standby_lag = 0` is not a readiness signal for promotion. If a "this context will survive validation" signal is feasible, it would be far more useful than lag, since it is the thing that actually determines whether the cache survives a failover.
Contributor guide
Research direction
Start with MasterService::RestoreFromStandbySnapshot in master_service.cpp around lines 2990, 3139, and 3193-3202, then inspect the failure handling in master_service_supervisor.cpp around line 401. Reproduce promotion with the supplied HA and eviction configuration and examine the validation failure and descriptor state. Done means valid objects survive restore, rejected descriptors are diagnosable, and restore failures and accepted/rejected objects are observable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, docker, kubernetes
- Domain
- backend, distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100