apache / apache/kvrocks

After CLUSTERX SETNODES, nodes assigned the slave role do not start replicating from the master

Open
#3,497 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
4.4k
Forks
658
Avg merge
1d 20h
Merged PRs (30d)
10

Description

## Summary

When a cluster is created or updated through the controller, the controller
pushes a `CLUSTERX SETNODES` payload that includes per-node master/slave
role assignments. Each node accepts the command (`+OK`), **but the nodes
marked as slave do not initiate replication** — `INFO replication` on every
node returns `role:master` with `connected_slaves:0`.

The replication link is never set up; every member behaves as a standalone
master that just happens to know about other members.

## Observed

Immediately after a fresh `POST /api/v1/namespaces/.../clusters` (54 nodes
with `replicas=3`, expected 18 shards × 1 master + 2 slaves), we sampled
each node's actual `INFO replication` `role:` and compared with the role
recorded in the controller meta:

```
Role alignment: match=26, mismatch=28, dead=0 (total 54)
```

Roughly half the nodes (28/54) are not following the master the controller
told them to follow. This shows up immediately after the API call — no
failure injection, no restart needed.

Once a node is in this "ghost master" state, no further controller action
(`SETNODES` re-push, `DELETE`+recreate, manager restart) brings it back
into a replica role.

## Expected

When `CLUSTERX SETNODES` (or the controller's create/update API) marks a
node as a slave of a particular master, the node should issue an internal
`SLAVEOF ` to start replication, OR the
controller should explicitly issue `REPLICAOF` to each slave after the
topology push.

## Versions

- Apache Kvrocks 2.15.0
- `kvctl-server` v1.15.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace the controller's CLUSTERX SETNODES handling from the create/update API and follow the node-side role-assignment path. Compare the assigned roles with INFO replication and verify whether slaves receive and execute SLAVEOF or REPLICAOF for their designated master. Done means every assigned slave reports role:slave and connected replication links match the controller topology.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, redis
Domain
api, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.