hashicorp / hashicorp/consul

consul_serf_coordinate_adjustment_ms metric increases for multiple days, goes back down and repeats

Open
#11,442 2 comments 0 reactions 0 assignees View on GitHub
theme/telemetry type/question
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

We recently discovered a strange behavior where a particular metric `consul_serf_coordinate_adjustment_ms` for our Consul Servers remains high for a number of days, goes back down and repeats. This metric represents how much consul is adjusting each time it updates: [Github Link](https://github.com/hashicorp/serf/blob/7faa1b06262f70780c3c35ac25a4c96d754f06f3/serf/ping_delegate.go#L77-L80).

#### Snapshots of the Graphs for different Consul servers

![image](https://user-images.githubusercontent.com/90725982/139313880-f632edb2-c705-49be-aabe-134f324c5014.png)

![image](https://user-images.githubusercontent.com/90725982/139313971-35792c0a-7b35-4f25-b530-22b8e8ffdb97.png)

Is this the normal behavior? Any ideas why the update operation is spread out across multiple days?

Contributor guide

Open the contributing guide

Research direction

Start with the referenced serf/ping_delegate.go lines and the consul_serf_coordinate_adjustment_ms metric, then compare that behavior with the attached graphs from the Consul servers. Done means determining whether the repeated multi-day increase and decrease is expected and documenting the cause or confirming a bug.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.