kvcache-ai / kvcache-ai/Mooncake
[Bug]: [NoF] Remount after HA recovery may reallocate offsets referenced by restored replicas
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
### Problem Description
Based on the current main branch, Mooncake restores NoF replica descriptors during HA + OpLog recovery, including the endpoint, offset, size, and status.
The restored NoF replicas are associated with a `DummyBufferAllocator`. This dummy allocator preserves the descriptors and endpoint but does not maintain the actual allocation state.
When the same NoF namespace is subsequently re-registered or remounted, Mooncake creates a new real allocator without importing the restored NoF descriptors into it.
As a result, the new allocator may still consider offsets referenced by restored replicas to be free and allocate them to new objects.
### Potential Impact
Mooncake indexes metadata by `(tenant_id, user_key)` and does not check whether descriptors belonging to different keys reference the same physical range.
Different keys may therefore reference the same `(endpoint, offset range)`. A later write can overwrite the physical data of an older object while its metadata remains `COMPLETE` and readable. A subsequent `Get` may then return incorrect data, resulting in silent data corruption.
### Questions for the Community
1. Are NoF replicas expected to remain readable after HA promotion?
2. When a NoF namespace is remounted, should its real allocator be rebuilt from the restored descriptors?
3. If NoF HA recovery is not currently supported, should restored replicas or the remounted segment remain unavailable to prevent a fresh allocator from overwriting existing data?
### Before submitting...
- [x] Ensure you searched for relevant issues and read the [documentation]
Contributor guide
Research direction
Start by tracing NoF replica descriptor restoration during HA and OpLog recovery, the DummyBufferAllocator, and real allocator creation during namespace re-registration or remount. Done requires resolving whether restored replicas remain readable and ensuring remounting cannot reallocate offsets they reference, or keeping unsupported recovered data unavailable to prevent corruption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100