Add a sync status endpoint / metrics to agents
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
### Background
The Consul agents sync to servers model is eventually consistent. It will not return errors when an action is performed but rather retry periodically :
* A service register call will always succeed even if the sync fails (we saw that with ACLs pre 0.8 : https://github.com/hashicorp/consul/pull/4771 and agent wipe : https://github.com/hashicorp/consul/pull/5217)
* A node id conflict can prevent a node from registering itself in the catalog
### Issue
Whether the sync failing is due to a bug, a bad setup, or temporary failure it is hard do detect these desyncs at scale :
* When such issues occur the only traces are in the logs
* There is no way to have an overview of an agent sync state
We would like to detect these issues to :
* Monitor the sync status of agents and their services
* Fail early in the case of conflicts
### Proposal
* Add an endpoint such as `/v1/agent/status` returning the state of the agent sync for its node, service , checks. Additionally other things such as the proposed async persistent state (https://github.com/hashicorp/consul/pull/6101) could find there place there too.
* Add metrics for these async elements to enable global monitoring
Contributor guide
Research direction
Start by reviewing the proposed /v1/agent/status endpoint and the existing agent sync behavior described in the issue, including its log traces and service, node, and check state. Done would require an agreed design for reporting sync status and metrics that supports monitoring agents and detecting conflicts; no files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100