After CLUSTERX SETNODES, nodes assigned the slave role do not start replicating from the master
- Dominant language
- C++
- Stars
- 4.4k
- Forks
- 658
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 10
Description
## Summary
When a cluster is created or updated through the controller, the controller
pushes a `CLUSTERX SETNODES` payload that includes per-node master/slave
role assignments. Each node accepts the command (`+OK`), **but the nodes
marked as slave do not initiate replication** — `INFO replication` on every
node returns `role:master` with `connected_slaves:0`.
The replication link is never set up; every member behaves as a standalone
master that just happens to know about other members.
## Observed
Immediately after a fresh `POST /api/v1/namespaces/.../clusters` (54 nodes
with `replicas=3`, expected 18 shards × 1 master + 2 slaves), we sampled
each node's actual `INFO replication` `role:` and compared with the role
recorded in the controller meta:
```
Role alignment: match=26, mismatch=28, dead=0 (total 54)
```
Roughly half the nodes (28/54) are not following the master the controller
told them to follow. This shows up immediately after the API call — no
failure injection, no restart needed.
Once a node is in this "ghost master" state, no further controller action
(`SETNODES` re-push, `DELETE`+recreate, manager restart) brings it back
into a replica role.
## Expected
When `CLUSTERX SETNODES` (or the controller's create/update API) marks a
node as a slave of a particular master, the node should issue an internal
`SLAVEOF ` to start replication, OR the
controller should explicitly issue `REPLICAOF` to each slave after the
topology push.
## Versions
- Apache Kvrocks 2.15.0
- `kvctl-server` v1.15.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Trace the controller's CLUSTERX SETNODES handling from the create/update API and follow the node-side role-assignment path. Compare the assigned roles with INFO replication and verify whether slaves receive and execute SLAVEOF or REPLICAOF for their designated master. Done means every assigned slave reports role:slave and connected replication links match the controller topology.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, redis
- Domain
- api, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100