cockroachdb / cockroachdb/cockroach
gossip: loss of incoming connections after large change
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
During a large cluster change (~90 nodes added to a large cluster), the gossip incoming connection count on two existing nodes dropped to zero and the nodes stopped getting new gossip updates. This persisted for ~2 hours until the 2 nodes were restarted, and then they were healthy again.
**To Reproduce**
We don't have exact steps to reproduce this. Likely steps:
Create a large cluster (>200 nodes)
Add another 100 nodes to the cluster
View the statistics for `gossip.connections.incoming` - it should stay above 0 on all nodes in the cluster, but it will drop on some nodes to 0 either for short or extended periods.
**Expected behavior**
`gossip.connections.incoming` should never stay at 0.
**Environment:**
- CockroachDB version 22.2 - but no recent change in this area.
**Additional context**
CRDB is very dependent on timely gossip updates. Not having gossip updates means liveness and store descriptor updates no longer happen. This makes the node assume all the other nodes in the cluster are dead. It then starts reporting under replicated and unavailable ranges and attempting to rebalance replicas around.
Jira issue: CRDB-32708
gz#18895
Contributor guide
Assessment
This issue has not been assessed yet.