cockroachdb / cockroachdb/cockroach

gossip: loss of incoming connections after large change

Open
#113,011 0 comments 0 reactions 0 assignees View on GitHub
A-kv-gossip C-bug O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

During a large cluster change (~90 nodes added to a large cluster), the gossip incoming connection count on two existing nodes dropped to zero and the nodes stopped getting new gossip updates. This persisted for ~2 hours until the 2 nodes were restarted, and then they were healthy again.

**To Reproduce**

We don't have exact steps to reproduce this. Likely steps:
Create a large cluster (>200 nodes)
Add another 100 nodes to the cluster
View the statistics for `gossip.connections.incoming` - it should stay above 0 on all nodes in the cluster, but it will drop on some nodes to 0 either for short or extended periods.

**Expected behavior**
`gossip.connections.incoming` should never stay at 0.

**Environment:**
- CockroachDB version 22.2 - but no recent change in this area.

**Additional context**
CRDB is very dependent on timely gossip updates. Not having gossip updates means liveness and store descriptor updates no longer happen. This makes the node assume all the other nodes in the cluster are dead. It then starts reporting under replicated and unavailable ranges and attempting to rebalance replicas around.

Jira issue: CRDB-32708

gz#18895

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.