cockroachdb / cockroachdb/cockroach

gossip: reduce store descriptor traffic amount

Open
#117,393 0 comments 0 reactions 0 assignees View on GitHub
C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
In large clusters, the gossip traffic can make up a significant portion of the total network traffic. We have seen clusters where it is >20% of total traffic (200 nodes with 10 disks/ node). We gossip the stores every 10 seconds and on a 200 node cluster this can be ~6 hops each. Each store desciptor is ~1KB since it includes a number of fields including a NodeDescriptor.

So in total this is:
`200 sending nodes * 10 disks/node * 200 receiving nodes * 6 hops * 1KB * 1/10 sec = 240MB/s `
in total of gossip traffic on the cluster.

In reality this traffic estimate might be an underestimate because `asyncGossipStore` is also called on lease and capacity changes and on a large cluster this can happens frequently.

**Describe the solution you'd like**
Most of the store descriptor traffic is immutable after startup, so the re-broadcasting of it is unnecessary. Additionally, the node descriptor being embedded in the store descriptor means we re-broadcast it more than necessary. We could likely broadcast just the mutable gossip traffic which is likely only 1/20 of the total traffic at a 10 second (or higher) frequency.

**Describe alternatives you've considered**
One alternative to this is to use a different point-to-point communication for relevant gossip information.
Another alternative is to make gossip more locality aware which reduces the cross-region traffic, but not the total traffic.
Another alternative is to aggregate up information by locality (so for instance store level, node level, rack level, AZ level, region level, ...) and not transmit this across the system.
Finally we currently require "full-knowledge" of the state of every other node within the allocator, but this isn't strictly necessary if we implemented something like copy-sets.

Many of these options are not mutually exclusive and implementing many of them will reduce gossip traffic further.

Jira issue: CRDB-35148

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.