cockroachdb / cockroachdb/cockroach

gossip: culling causes instability in some topologies

Open
#133,381 1 comment 0 reactions 0 assignees View on GitHub
A-kv-gossip branch-release-23.1 C-bug O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

There are certain gossip network topologies where the culling mechanism of removing the least useful client can lead to repeated connecting and disconnecting of the gossip network.

Consider the gossip network below where `n1` is the leaseholder for range 1 (meta1).
Screenshot 2024-10-24 at 12 33 06 PM

If `n3` is restarted (e.g. as part of a rolling restart), the gossip network is disconnected. Nodes 1 and 2 will not initiate any new client connections because they have one already and because their gossip sentinel TTLs have not expired (`n1` continues to gossip the sentinel key).

The rest of the nodes in the network see an expired gossip sentinel and start establishing new connections. Fairly quickly one of them, say `n`, connects to `n1` or `n2`, and gossip recovers. With a sufficiently large network, `n` has likely established other connections before landing on `n1` or `n2`, so it's quite likely that it has hit the max number of outgoing connections and will try to do some culling. This is even more likely if `n1` and `n2` are towards the end of the list of bootstrap addresses (which is not randomized).

[Culling](https://github.com/cockroachdb/cockroach/blob/master/pkg/gossip/gossip.go#L1348) involves removing a connection to the [least useful](https://github.com/cockroachdb/cockroach/blob/master/pkg/gossip/infostore.go#L518) node, defined as the node that contributes infos from the fewest originating nodes. If `n` connected to `n1`, it will see a single contribution from `n1` (`n1` itself); if it connected to `n2`, it will see two contributions from `n2` (`n1` and `n2`). In either case, the new connection is likely to be culled since `n` is well-connected to the rest of the network and its other peers contribute more. This repeats the process of disconnecting and reconnecting gossip, until the range 1 lease moves elsewhere (are there other ways the network can recover on its own?).

[Tightening](https://github.com/cockroachdb/cockroach/blob/master/pkg/gossip/gossip.go#L1397) the network could help here, if either `n1` and `n2` or other nodes try to establish a connection to the far end of the network. But in this minimal example, `n1` could be only a couple of hops away from the well-connected part of the network, so tightening will not be enabled. This will probably vary based on the size of the network and the value of `maxHops`.

This topology seems very rare (hence the `P3` priority). We've seen it in the wild during a rolling restart. We should consider whether rolling restarts make this more likely; e.g. because nodes' lists of bootstrap addresses are not randomized, or because nodes leave and join one by one (potentially in the same order as the bootstrap address list).

Jira issue: CRDB-43562

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.