[RFC]: Hot Standby Mode for Master Service Metadata High Availability

## Introduction

To address the issue where all metadata is lost after a master service failure and recovery, this solution proposes a **hot-standby synchronization mechanism** and a **hybrid approach** combining hot standby with snapshots, ensuring rapid data recovery following a master service failure.

## Motivation

The current HA (High Availability) mode only supports rapid restart of the master service after a failure but cannot guarantee fast data recovery. The pain points of the current HA mode are as follows:

| Issue | Impact | Severity |
| -------------------------------------------------- | ------------------------------------------------------------ | ---------- |
| **Complete loss of metadata after Leader failure** | All KV cache location information is lost and must be rebuilt | 🔴 Critical |
| **Follower lacks data pre-warming** | After successful election, the new Leader must start serving from scratch | 🟠 Medium |
| **Long recovery time** | Relies on clients to re-register; may take several minutes to tens of minutes | 🟠 Medium |
| **Service unavailability window** | Requests cannot be processed during Leader failover | 🟡 Low |

## Goals:

1. **Near-zero RPO**: Minimize the data loss window
2. **Fast failover**: Achieve second-level RTO (Recovery Time Objective)
3. **Data consistency guarantee**: Ensure correct synchronization between primary and standby
4. **Verifiability**: Include mechanisms to validate data integrity
5. **Low performance overhead**: Avoid impacting normal business operations

## Proposal

### 1. Core Design Approach

**Use etcd as an intermediate reliability component to implement OpLog primary-standby synchronization**:

- **OpLog Mechanism**: Primary Master records all state change operations to OpLog
- **etcd Storage**: OpLog is written to etcd, leveraging etcd's strong consistency and persistence capabilities
- **Watch Mechanism**: Standby Master receives OpLog in real-time through etcd Watch mechanism
- **Ordering Guarantee**: Guarantee operation order through global sequence_id and key-level key_sequence_id

### 2. Architecture Design

#### (1) Overall Architecture

The overall architecture diagram shows the interaction relationships between Primary Master, etcd Cluster, and Standby Master:

**Architecture Description**:

- **Primary Master**: Handles client requests, records OpLog and writes to etcd
- **etcd Cluster**: Acts as intermediate storage, providing strong consistency and Watch mechanism
- **Standby Master**: Receives OpLog in real-time by watching etcd and applies to local metadata store

#### (2) Data Flow Diagram

The data flow diagram shows the complete flow from Client request to Standby synchronization:

**Flow Description**:

a. Client sends `PutEnd` request to Primary Master
b. Primary Master records operation through `OpLogManager`, generating sequence_id
c. `EtcdOpLogStore` writes OpLog to etcd
d. etcd notifies Standby Master through Watch mechanism
e. `OpLogWatcher` receives events and passes to `OpLogApplier`
f. `OpLogApplier` checks order and applies to Standby's metadata store

#### (3) Failover Sequence

The failover sequence diagram shows the complete process from Primary failure to Standby promotion to Primary:

**Flow Description**:

a. **Normal Operation**: Primary maintains Lease, Standby continuously synchronizes OpLog through Watch
b. **Primary Failure**: Primary's Lease expires, etcd notifies Standby
c. **Standby Promotion**: Stop Standby service, initialize Lease, clean expired metadata, start leader election

### 3. Core Component Design

#### (1) OpLogManager (Primary Side)

**Responsibilities**:

- Record all state change operations (PUT_END, PUT_REVOKE, REMOVE)
- Generate global sequence_id and key-level key_sequence_id
- Maintain memory buffer (for fast queries)

**Key Methods**:

```cpp
class OpLogManager {
uint64_t Append(OpType type, const std::string& key,
const std::string& payload = "");
std::vector GetEntriesSince(uint64_t since_seq_id,
size_t limit = 1000) const;
uint64_t GetLastSequenceId() const;
};
```

#### (2) EtcdOpLogStore (Primary Side)

**Responsibilities**:

- Write OpLog to etcd
- Update latest sequence_id
- Record snapshot corresponding sequence_id
- Clean up old OpLog

**etcd Key Design**:

- OpLog Entry: `mooncake-store/oplog/{cluster_id}/{sequence_id}`
- Latest Sequence ID: `mooncake-store/oplog/{cluster_id}/latest`
- Snapshot Sequence ID: `mooncake-store/oplog/{cluster_id}/snapshot/{snapshot_id}/sequence_id`

#### (3) OpLogWatcher (Standby Side)

**Responsibilities**:

- Watch etcd OpLog changes
- Read historical OpLog (for initial synchronization)
- Process Watch events and pass to OpLogApplier

**Key Methods**:

```cpp
class OpLogWatcher {
void Start();
void Stop();
bool ReadOpLogSince(uint64_t start_seq_id,
std::vector& entries);
};
```

#### (4) OpLogApplier (Standby Side)

**Responsibilities**:

- Apply OpLog Entry to local metadata store
- Check global and key-level order
- Handle sequence number discontinuities and out-of-order cases
- Periodically clean up key_sequence_map_ (memory optimization)

**Key Methods**:

```cpp
class OpLogApplier {
bool ApplyOpLogEntry(const OpLogEntry& entry);
bool CheckSequenceOrder(const OpLogEntry& entry);
void CleanupStaleKeySequences();
};
```

#### (5) HotStandbyService (Standby Side)

**Responsibilities**:

- Manage Standby mode lifecycle
- Coordinate OpLogWatcher and OpLogApplier
- Handle Standby promotion to Primary logic

**Key Methods**:

```cpp
class HotStandbyService {
void StartStandby();
void Stop();
void Promote();
};
```

### 4. OpLog Entry Data Structure

```cpp
struct OpLogEntry {
uint64_t sequence_id{0}; // Globally monotonically increasing sequence
uint64_t timestamp_ms{0}; // Timestamp (milliseconds)
OpType op_type{OpType::PUT_END}; // PUT_END, PUT_REVOKE, REMOVE
std::string object_key; // Object key
std::string payload; // Optional payload (carries replica info for PUT_END)
uint32_t checksum{0}; // Checksum
uint32_t prefix_hash{0}; // Key prefix hash
uint64_t key_sequence_id{0}; // Per-key operation sequence (for ordering guarantee)
};
```

**JSON Serialization Format**:

```json
{
"sequence_id": 12345,
"timestamp": 1704110400123,
"op_type": "PUT_END",
"key": "object_key_123",
"payload": "optional_payload",
"checksum": 1234567890,
"prefix_hash": 987654321,
"key_sequence_id": 5
}
```

### 5. Ordering Guarantee Mechanism

#### (1) Global Sequence Number (sequence_id)

- **Purpose**: Guarantee global order of all OpLog events
- **Generation**: Generated globally incrementally by Primary's `OpLogManager`
- **Check**: Standby checks if sequence_id is continuous

#### (2) Key-Level Sequence Number (key_sequence_id)

- **Purpose**: Guarantee operation order for the same key
- **Generation**: Incremented separately for each key on Primary side
- **Check**: Standby checks if key_sequence_id is increasing

#### (3) Out-of-Order Handling

When key_sequence_id out-of-order is detected:

- **Rollback**: Delete all state of the key from metadata_store
- **Replay**: Re-read all OpLog from etcd starting from the key's first sequence_id
- **Rewrite**: Re-apply all OpLog in correct order to rebuild metadata

### 6. Snapshot Integration

#### (1) Record Sequence ID During Snapshot

- When snapshot is generated, record current OpLog sequence_id
- Write snapshot info to etcd: `mooncake-store/oplog/{cluster_id}/snapshot/{snapshot_id}/sequence_id`

#### (2) OpLog Cleanup

- After snapshot generation, OpLog before snapshot can be cleaned up
- Cleanup strategy: Query minimum existing sequence_id from etcd, use DeleteRange to delete

### 7. Standby Service Integration

#### (1) Problem

In existing code, Standby only blocks and waits during leader election, without running Standby service to synchronize OpLog.

#### (2) Solution

In `MasterServiceSupervisor::Start()`:

- Check if there is currently a leader
- If there is a leader and it's not self → Start Standby service (watch OpLog and apply)
- After successful election → Stop Standby service and promote to Primary

### 8. Lease Initialization When Standby Promotes to Primary

#### (1) Problem

Objects on Standby all have lease = 0 (because OpLog only contains PUT_END, not renewal information), and all objects will expire immediately after promotion to Primary.

#### (2) Solution

In `HotStandbyService::Promote()`:

- Stop Standby service
- Iterate through all metadata
- For objects with lease_timeout = 0, grant default lease time (`default_kv_lease_ttl`)

### 9. Memory Optimization: key_sequence_map_ Cleanup

#### (1) Problem

`key_sequence_map_` on Standby side is used to track `key_sequence_id` for each key. After metadata is deleted, these entries are still retained, which may cause memory leaks during long-term operation.

#### (2) Solution

Implement periodic cleanup mechanism:

- **Cleanup Condition**: Last operation is `REMOVE` and more than 1 hour has passed
- **Cleanup Frequency**: Scan once per hour
- **Retention Strategy**: Keys with `PUT_END` and `PUT_REVOKE` operations are not cleaned

## Key Design Points Summary

1. **etcd as Intermediate Storage**: Leverage etcd's strong consistency and Watch mechanism
2. **Record Only Critical Operations**: PUT_END, PUT_REVOKE, REMOVE, do not record LEASE_RENEW
3. **Dual Sequence Number Guarantee**: Global sequence_id + key-level key_sequence_id
4. **Snapshot Integration**: Integrate with existing snapshot mechanism, support OpLog cleanup
5. **Standby Service Runs in Parallel**: Continuously synchronize data during election waiting period
6. **Memory Optimization**: Periodically clean up expired entries in key_sequence_map_

### Before submitting a new issue...

- [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start by reading MasterServiceSupervisor::Start() and the proposed HotStandbyService, OpLogManager, OpLogWatcher, and OpLogApplier responsibilities. Trace the existing leader-election and snapshot flows before deciding how the proposed etcd synchronization fits. Done means the design is resolved into an implementable change that covers synchronization, promotion, ordering, snapshots, and key_sequence_map_ cleanup.

Written by the indexing model from the issue text.