apache / apache/kvrocks

Proposal for Handling Orphaned Slaves in Cluster Mode

Open
#2,841 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
4.4k
Forks
658
Avg merge
1d 20h
Merged PRs (30d)
10

Description

### Search before asking

- [ ] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.

### Motivation

Currently, there is a minor issue in Kvrocks cluster mode: When a slave node is deleted via the controller, the controller only notifies this change to nodes within the cluster. As a result, the deleted slave remains unaware of its removal and continues replication workflows, which is counterintuitive. This proposal aims to resolve this issue.

#### Key Considerations:
1. **Unreliable Controller-to-Node Notification**:
Relying on the controller to synchronize deletion events to the target node is unreliable. When a node is deleted, it often indicates an unhealthy state (e.g., network instability). We cannot guarantee successful delivery of deletion notifications (e.g., messages may arrive but fail to return acknowledgments). Thus, unlimited retries on the controller side are not a robust design.

2. **Protocol Flexibility**:
While Kvrocks' master-slave replication protocol is inspired by Redis, it is not identical. Introducing additional replication steps distinct from Redis is acceptable.

3. **Topology Synchronization Priority**:
The controller does not prioritize updating topology information for the current shard’s master. Consequently, masters must account for potentially stale topology data when validating slave replication requests and avoid outright rejection.

### Solution

1. **Slave Metadata Propagation**:
During `replconf` execution, slaves will send their **node ID** and **current config version** to the master. The master will store this metadata.

2. **Version-Based Validation**:
After processing `replconf` or `cluster setnodes` commands, the master will scan connections meeting the following criteria and return a specific error to trigger replication termination on the slave side:
- **Condition**: The master’s topology version is newer than the slave’s (`master_version > slave_version`), **and** the slave’s node ID no longer exists in the master’s topology.

3. **Cleanup Workflow**:
Slaves receiving the specific error will terminate replication and clean up their state.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the replconf and cluster setnodes command handlers, then follow how replication connections store state and handle errors. Confirm how topology versions and slave node IDs are represented before assessing the cleanup path. Done means an orphaned slave receives the specific termination signal and cleans up replication state when its node is absent from newer topology.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, redis
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.