apache / apache/kvrocks-controller

Controller's cluster_nodes push silently drops when a data node is unreachable, leaving cluster meta stale forever

Open
#395 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
141
Forks
76
PR merge metrics
No merged PRs in 30d

Description

## Summary

`kvctl-server` propagates cluster topology via a push to each data node, but
if the push fails (network blip, target restarting, etc.) it is **not retried**.
After a controller restart or any partial failure the data nodes end up with
different views of the cluster, and the cluster does not self-heal.

## Observed

In an 18-shard cluster (`replicas=3`, 54 nodes) we ran `CLUSTER NODES` on
four representative masters at the same time:

| Master | Visible nodes (master + slave) |
|--------|--------------------------------|
| shard 0 | **3** (a residual view from a 3-shard cluster created earlier) |
| shard 1 | 17 |
| shard 9 | 33 (a residual view from a 27-shard cluster created in between) |
| shard 17 | 8 |
| Expected | **54** |

`cluster_state:ok` and `cluster_slots_ok:16384` on every node, so routing
formally works, but every node holds a different `cluster_nodes` snapshot
and clients that bootstrap from a "small" master keep redirecting traffic
to a few hot masters.

The state never converges back. Manager restart (rolling, all 3 instances)
only partially refreshed the views. There is no operator-facing endpoint
(`/sync`, `/refresh`, `/reload`, …) to force a re-push — all returned 404.

## Expected

* Controller retries the per-node push with backoff until each node
acknowledges the new topology version, OR
* Exposes a `POST /clusters/{name}/sync` (or similar) endpoint that
re-pushes the current `cluster_nodes` to every member.

## Repro (vanilla setup)

1. Start `kvctl-server` v1.3.0 and 6 Apache Kvrocks 2.15.0 data nodes.
2. Create a cluster with `replicas=2` (2 shards × 3 nodes).
3. `iptables -A INPUT -p tcp --dport -j DROP` for ~10 s.
4. Remove the rule, wait, then run `CLUSTER NODES` on every node.

Node 3 has a stale `cluster_nodes` view; nothing converges back even
after several minutes.

## Versions

- `kvctl-server` v1.3.0
- Apache Kvrocks 2.15.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.