kvcache-ai / kvcache-ai/Mooncake

[RFC][Store] Distinguish new segment registration from segment remount

Open
#3,560 2 comments 1 reaction 0 assignees View on GitHub
RFC
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Changes proposed

## Motivation

This follows up on #3517 and provides context for #3533.

Currently, `ReMountSegment()` is used in more than one situation:

1. A client receives `NEED_REMOUNT` from `Ping()` and re-registers its existing segments after a Master failover or loss of Master-side state.
2. Newly added segments—including segments exposed by newly added P nodes—may also go through the same RPC path for initial registration.

Therefore, receiving a `ReMountSegment()` request does not necessarily mean that standby recovery is required.

The Master currently cannot reliably distinguish a fresh segment registration from the remount of a segment that existed before the failover.

## Problem

The current RPC semantics overload initial segment registration and segment recovery into the same operation.

In particular, standby recovery logic may interpret a `ReMountSegment()` request as evidence that the segment should already exist in the restored Master state. However, the request may contain a newly added segment for which no previous Master-side state exists.

This ambiguity couples the segment lifecycle with the client's remount status and makes the recovery path harder to reason about. It may also cause newly registered segments to enter unnecessary standby-recovery logic, including metadata scans intended only for previously known segments.

## Proposed direction

Instead of inferring the operation solely from the use of `ReMountSegment()` or from the client's remount status, explicitly propagate each segment's lifecycle status or registration intent to the Master.

For example, a segment update could distinguish between states such as:

- `NEW`: A newly registered segment that has no previous Master-side state and does not require standby recovery.
- `REMOUNT`: An existing segment being re-registered after a Master failover or loss of Master-side state.
- `STALE`: A segment whose previous Master-side state should no longer be considered active.

The exact naming, state transitions, and wire format are open for discussion. We should also determine whether `STALE` belongs to the same update protocol or should be represented by a separate removal or synchronization operation.

The main goal is to make the segment lifecycle explicit instead of deriving it indirectly from the RPC being used or from client-level remount status.

With explicit per-segment intent, the Master can run standby recovery only for segments marked as `REMOUNT`, while treating `NEW` segments as ordinary registrations. This may also allow the current client remount status to be simplified or redefined later.

## Expected benefits

1. Avoid treating newly added segments as standby-recovery segments.
2. Skip full metadata scans when a request contains no segments requiring standby recovery.
3. Separate segment lifecycle from client liveness and remount readiness.
4. Make failover and standby recovery behavior easier to reason about.
5. Reduce special cases in and around `ReMountSegment()`.
6. Provide a cleaner foundation for future segment-state synchronization between clients and the Master.

## Context

This came up while discussing standby recovery behavior. A `ReMountSegment()` call alone is not sufficient to determine whether recovery of previous Master-side segment state is expected, because the same request path may also be used to register newly added segments.

@Aionw

### Before submitting a new issue...

- [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the ReMountSegment() and Ping() paths described in the issue, including how the Master handles standby recovery and newly registered segments. Clarify the per-segment lifecycle states, their transitions, and wire representation; done means NEW registrations avoid recovery, REMOUNT segments trigger it, and the treatment of STALE segments is defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.