kvcache-ai / kvcache-ai/Mooncake

Regarding Mooncake's HA (High Availability) capability, there are still some issues to be resolved as of now.

Open
#3,561 2 comments 0 reactions 0 assignees View on GitHub
RFC
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

While reviewing the code on the main branch, we analyzed Mooncake's HA capability and identified several outstanding issues:

1. The oplog only supports etcd mode, and each time it must perform a cold start from seq 0. When the oplog volume is large, this becomes a serious problem.
2. Due to the issue above, there is currently no GC (garbage collection) mechanism for oplog data in etcd.
3. The snapshot is only read once during standby initialization and is never read again afterward. If a failover occurs, this will cause a significant gap between the standby and the leader.
4. Currently, snapshot and oplog operate in a mutually exclusive (either/or) mode. When both are enabled simultaneously, the leader/master will stop writing snapshots entirely.
Could the community share its specific plans to address these issues? At this point, it doesn't appear that the current HA implementation offers a robust solution to mitigate these shortcomings. We would greatly appreciate any feedback or insights!

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the HA implementation on the main branch and the linked Mooncake documentation, focusing on oplog recovery, oplog garbage collection, standby snapshot refresh, and simultaneous snapshot/oplog operation. Done would require an agreed plan and implementation that addresses all four reported limitations, but the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.