consul_serf_coordinate_adjustment_ms metric increases for multiple days, goes back down and repeats
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
We recently discovered a strange behavior where a particular metric `consul_serf_coordinate_adjustment_ms` for our Consul Servers remains high for a number of days, goes back down and repeats. This metric represents how much consul is adjusting each time it updates: [Github Link](https://github.com/hashicorp/serf/blob/7faa1b06262f70780c3c35ac25a4c96d754f06f3/serf/ping_delegate.go#L77-L80).
#### Snapshots of the Graphs for different Consul servers


Is this the normal behavior? Any ideas why the update operation is spread out across multiple days?
Contributor guide
Research direction
Start with the referenced serf/ping_delegate.go lines and the consul_serf_coordinate_adjustment_ms metric, then compare that behavior with the attached graphs from the Consul servers. Done means determining whether the repeated multi-day increase and decrease is expected and documenting the cause or confirming a bug.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100