Support clients to access kvrocks cluster using LB
- Dominant language
- C++
- Stars
- 4.4k
- Forks
- 658
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 10
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.
### Motivation
In the following situations, a master-slave switchover in a Kvrocks cluster may cause a split-brain situation:
1. The Kvrocks process hangs and then recovers
2. The network card on the Kvrocks host fails and then recovers
3. The network partition on the Kvrocks host recovers
When the Kvrocks process regains access, client requests may be directed to the old master node for a brief period, causing a split-brain situation and data inconsistency.
If clients access Kvrocks using a load balancer (LB), when a Kvrocks cluster switchover occurs, redirecting the LB address to the new master node ensures that client requests are not forwarded to the old master node.
The Kvrocks cluster deployment architecture is as follows:
Benefits of this architecture:
1. It covers most split-brain scenarios, including process restarts, process hangs, host network interface downtime, and network partitions.
2. Using a load balancer (LB) accelerates failover, significantly faster than approaches such as using dynamic domain names on the client side.
This approach is used in both AWS Elasticache and Azure Redis.
### Solution
To ensure that client access to the KvRocks cluster goes through the load balancer (LB), while maintaining local IP for internal KvRocks cluster communication (master-slave synchronization and slot migration), the following modifications are required:
1. Add two optional parameters to the clusterx setnodes command to specify a CLB host for each node.
The following example shows that you can specify some or all LBs:
CLUSTERX SETNODES "ZrFserb4Mqi5dbyCLCMUm9zFXMNPhzKE4RbFauJa 10.190.28.10 7115 master - 0-5460 \n qhbVLW9dc2VF9CZOmvBJOzzqkhNA3oHoD3NVWVyg 10.190.28.10 7133 master - 5461-10921 \n 9cnjXnaNK5KfPtX01V0wn7ZtM7FDPnz7va9qPJBs 10.190.28.10 7237 master - 10922-16383 **192.168.0.2 1222**\n fuCHh8Ru9tFSY313QPfBcPQ3QeP63Z4rWpqpyaOl 10.190.28.10 7255 slave ZrFserb4Mqi5dbyCLCMUm9zFXMNPhzKE4RbFauJa \n cIqdj5C37cMrr7r7ydZDZ8zm1T1vPLMWj8516rrF 10.190.28.10 6741 slave qhbVLW9dc2VF9CZOmvBJOzzqkhNA3oHoD3NVWVyg \n 3D8JRN8YVONqRU57jP99E3JdH2QnLHzZpFN4rcQQ 10.190.28.10 6765 slave 9cnjXnaNK5KfPtX01V0wn7ZtM7FDPnz7va9qPJBs **192.168.0.1 1234**" 1754120940
2. When persisting nodes.conf to a local file, add the LB address information to the file. When the kvrovks process starts, it will also load the corresponding LB address information.
3. When executing the cluster nodes/slots/replicas commands, if the nodes have a LB host configured, replace it with LB host.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the CLUSTERX SETNODES command and the nodes.conf persistence and loading paths. Then inspect the cluster nodes, slots, and replicas command handling. Done means optional LB addresses persist across restarts, client-facing cluster responses use them, and internal synchronization and slot migration retain local IPs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, redis
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100