cockroachdb / cockroachdb/cockroach

kvserver: add a gossip histogram metrics for latency

Open
#125,443 0 comments 0 reactions 0 assignees View on GitHub
A-kv-gossip C-enhancement O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
Timely gossip delivery is important for liveness and rebalancing decisions. In some cases we have either seen slow gossip propagation as a problem causing liveness failures. Having a histogram of the delay between when a gossip was first published, `OrigStamp`, and when it was added to the `infoStore` will give us insight into these delays. Additionally in cross-regional networks gossip messages often have to cross regions multiple times to get from the source to the destination. For a large cross-region cluster this could result in unacceptable delays for time-sensitive messages.

**Describe the solution you'd like**
A histogram which show the time difference between when an info is originally created vs when it was added to a store in `addInfo` would help debug complex gossip network delays. There is a separate histogram already added as part of #125148 for the callback processing delay, but we also care about the propagation delay.

Jira issue: CRDB-39458

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.