cockroachdb / cockroachdb/cockroach

server: improve clock offset monitoring

Open
#114,321 0 comments 0 reactions 0 assignees View on GitHub
A-cluster-observability C-enhancement O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The `Clock Offset` graph in DB console displays a [mean](https://github.com/cockroachdb/cockroach/blob/2ffa52615afb128697175762761800f8974c9eda/pkg/rpc/clock_offset.go#L342-L351) offset from one node to other nodes. The offsets are signed, so it's possible to distinguish a node's clock that is mostly behind or mostly in front of other nodes. Example:

Screenshot 2023-10-02 at 17 05 38

The **mean** offset is not necessarily the best metric for analysis, for reasons:
- positive and negative offsets cancel each other out
- one skewed node messes up all nodes' offset graphs, which makes it harder to identify the outlier

We should have more comprehensive metrics.

1. For example, in addition to the mean offset, we could report a histogram, or at least a set of: min offset, max offset, 50%.

2. Also, the number of nodes participating in the computation can [change](https://github.com/cockroachdb/cockroach/blob/2ffa52615afb128697175762761800f8974c9eda/pkg/rpc/clock_offset.go#L339) dynamically. We could plot this figure as well.

3. A node can [terminate](https://github.com/cockroachdb/cockroach/blob/82d57212ee1c4e256171545d7ae02bbb5117b8b9/pkg/rpc/peer.go#L424-L435) itself if its clock is a 50%+ outlier from other nodes. We should make metrics that are indicative of this event coming, so that alerting can notice this situation earlier than the node kills itself. That is why something like a 50%-ile offset graph is a better indicator. Another indicator could be: the number/percent of nodes whose offset is above the threshold.

Jira issue: CRDB-33459

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.