kvcache-ai / kvcache-ai/Mooncake
[RFC][Store] Distinguish new segment registration from segment remount
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Changes proposed
## Motivation
This follows up on #3517 and provides context for #3533.
Currently, `ReMountSegment()` is used in more than one situation:
1. A client receives `NEED_REMOUNT` from `Ping()` and re-registers its existing segments after a Master failover or loss of Master-side state.
2. Newly added segments—including segments exposed by newly added P nodes—may also go through the same RPC path for initial registration.
Therefore, receiving a `ReMountSegment()` request does not necessarily mean that standby recovery is required.
The Master currently cannot reliably distinguish a fresh segment registration from the remount of a segment that existed before the failover.
## Problem
The current RPC semantics overload initial segment registration and segment recovery into the same operation.
In particular, standby recovery logic may interpret a `ReMountSegment()` request as evidence that the segment should already exist in the restored Master state. However, the request may contain a newly added segment for which no previous Master-side state exists.
This ambiguity couples the segment lifecycle with the client's remount status and makes the recovery path harder to reason about. It may also cause newly registered segments to enter unnecessary standby-recovery logic, including metadata scans intended only for previously known segments.
## Proposed direction
Instead of inferring the operation solely from the use of `ReMountSegment()` or from the client's remount status, explicitly propagate each segment's lifecycle status or registration intent to the Master.
For example, a segment update could distinguish between states such as:
- `NEW`: A newly registered segment that has no previous Master-side state and does not require standby recovery.
- `REMOUNT`: An existing segment being re-registered after a Master failover or loss of Master-side state.
- `STALE`: A segment whose previous Master-side state should no longer be considered active.
The exact naming, state transitions, and wire format are open for discussion. We should also determine whether `STALE` belongs to the same update protocol or should be represented by a separate removal or synchronization operation.
The main goal is to make the segment lifecycle explicit instead of deriving it indirectly from the RPC being used or from client-level remount status.
With explicit per-segment intent, the Master can run standby recovery only for segments marked as `REMOUNT`, while treating `NEW` segments as ordinary registrations. This may also allow the current client remount status to be simplified or redefined later.
## Expected benefits
1. Avoid treating newly added segments as standby-recovery segments.
2. Skip full metadata scans when a request contains no segments requiring standby recovery.
3. Separate segment lifecycle from client liveness and remount readiness.
4. Make failover and standby recovery behavior easier to reason about.
5. Reduce special cases in and around `ReMountSegment()`.
6. Provide a cleaner foundation for future segment-state synchronization between clients and the Master.
## Context
This came up while discussing standby recovery behavior. A `ReMountSegment()` call alone is not sufficient to determine whether recovery of previous Master-side segment state is expected, because the same request path may also be used to register newly added segments.
@Aionw
### Before submitting a new issue...
- [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Contributor guide
Research direction
Start by tracing the ReMountSegment() and Ping() paths described in the issue, including how the Master handles standby recovery and newly registered segments. Clarify the per-segment lifecycle states, their transitions, and wire representation; done means NEW registrations avoid recovery, REMOUNT segments trigger it, and the treatment of STALE segments is defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100