Proposal for Handling Orphaned Slaves in Cluster Mode
- Dominant language
- C++
- Stars
- 4.4k
- Forks
- 658
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 10
Description
### Search before asking
- [ ] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.
### Motivation
Currently, there is a minor issue in Kvrocks cluster mode: When a slave node is deleted via the controller, the controller only notifies this change to nodes within the cluster. As a result, the deleted slave remains unaware of its removal and continues replication workflows, which is counterintuitive. This proposal aims to resolve this issue.
#### Key Considerations:
1. **Unreliable Controller-to-Node Notification**:
Relying on the controller to synchronize deletion events to the target node is unreliable. When a node is deleted, it often indicates an unhealthy state (e.g., network instability). We cannot guarantee successful delivery of deletion notifications (e.g., messages may arrive but fail to return acknowledgments). Thus, unlimited retries on the controller side are not a robust design.
2. **Protocol Flexibility**:
While Kvrocks' master-slave replication protocol is inspired by Redis, it is not identical. Introducing additional replication steps distinct from Redis is acceptable.
3. **Topology Synchronization Priority**:
The controller does not prioritize updating topology information for the current shard’s master. Consequently, masters must account for potentially stale topology data when validating slave replication requests and avoid outright rejection.
### Solution
1. **Slave Metadata Propagation**:
During `replconf` execution, slaves will send their **node ID** and **current config version** to the master. The master will store this metadata.
2. **Version-Based Validation**:
After processing `replconf` or `cluster setnodes` commands, the master will scan connections meeting the following criteria and return a specific error to trigger replication termination on the slave side:
- **Condition**: The master’s topology version is newer than the slave’s (`master_version > slave_version`), **and** the slave’s node ID no longer exists in the master’s topology.
3. **Cleanup Workflow**:
Slaves receiving the specific error will terminate replication and clean up their state.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the replconf and cluster setnodes command handlers, then follow how replication connections store state and handle errors. Confirm how topology versions and slave node IDs are represented before assessing the cleanup path. Done means an orphaned slave receives the specific termination signal and cleans up replication state when its node is absent from newer topology.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, redis
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100