kvcache-ai / kvcache-ai/Mooncake

[Bug]: [NoF] Remount after HA recovery may reallocate offsets referenced by restored replicas

Open
#3,826 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Bug Report

### Problem Description

Based on the current main branch, Mooncake restores NoF replica descriptors during HA + OpLog recovery, including the endpoint, offset, size, and status.

The restored NoF replicas are associated with a `DummyBufferAllocator`. This dummy allocator preserves the descriptors and endpoint but does not maintain the actual allocation state.

When the same NoF namespace is subsequently re-registered or remounted, Mooncake creates a new real allocator without importing the restored NoF descriptors into it.

As a result, the new allocator may still consider offsets referenced by restored replicas to be free and allocate them to new objects.

### Potential Impact

Mooncake indexes metadata by `(tenant_id, user_key)` and does not check whether descriptors belonging to different keys reference the same physical range.

Different keys may therefore reference the same `(endpoint, offset range)`. A later write can overwrite the physical data of an older object while its metadata remains `COMPLETE` and readable. A subsequent `Get` may then return incorrect data, resulting in silent data corruption.

### Questions for the Community

1. Are NoF replicas expected to remain readable after HA promotion?
2. When a NoF namespace is remounted, should its real allocator be rebuilt from the restored descriptors?
3. If NoF HA recovery is not currently supported, should restored replicas or the remounted segment remain unavailable to prevent a fresh allocator from overwriting existing data?

### Before submitting...

- [x] Ensure you searched for relevant issues and read the [documentation]

Contributor guide

Open the contributing guide

Research direction

Start by tracing NoF replica descriptor restoration during HA and OpLog recovery, the DummyBufferAllocator, and real allocator creation during namespace re-registration or remount. Done requires resolving whether restored replicas remain readable and ensuring remounting cannot reallocate offsets they reference, or keeping unsupported recovered data unavailable to prevent corruption.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.