apache / apache/kvrocks

Implement CLUSTER FAILOVER [FORCE|TAKEOVER] for manual master swap

Open
#3,496 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
4.4k
Forks
658
Avg merge
1d 20h
Merged PRs (30d)
10

Description

## Summary

Redis Cluster's `CLUSTER FAILOVER [FORCE|TAKEOVER]` lets an operator
promote a replica to master, either for planned maintenance or when the
current master is unreachable. Apache Kvrocks rejects the command:

```
> CLUSTER FAILOVER FORCE
-ERR CLUSTER command, CLUSTER INFO|NODES|SLOTS|KEYSLOT|RESET|REPLICAS

> CLUSTERX FAILOVER FORCE
-ERR CLUSTERX command, CLUSTERX VERSION|MYID|SETNODEID|SETNODES|SETSLOT|MIGRATE
```

So neither `CLUSTER` nor the kvrocks-specific `CLUSTERX` namespace
implements failover.

## Why this matters

Without `CLUSTER FAILOVER`, the only recovery path when a master is
stuck is to delete and recreate the cluster via the controller, which
is disruptive even when a fully synced replica is available.

In controller-managed deployments, the controller's automated failover
(ping-based) is the only mechanism — and it does not always trigger in
practice (e.g. we waited >2 min with `ping_interval_seconds=5` and
`min_alive_size=2` after a single master task restart and no promotion
happened).

## Expected

Implement Redis-compatible `CLUSTER FAILOVER [FORCE|TAKEOVER]`,
at minimum `TAKEOVER` for the operator-driven case where the existing
master is known to be unreachable. Both replicas and clients then have
a standards-compliant way to recover from a stuck master.

## Versions

- Apache Kvrocks 2.15.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the existing CLUSTER and CLUSTERX command entry points and the controller's automated failover path. Compare the supported Redis failover semantics for FORCE and TAKEOVER, then verify behavior with replica promotion and client routing scenarios; done means both namespaces expose the intended compatible command behavior without disrupting healthy replicas.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, redis
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.