hashicorp / hashicorp/consul

Add a sync status endpoint / metrics to agents

Open
#6,119 0 comments 0 reactions 0 assignees View on GitHub
theme/telemetry type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

### Background
The Consul agents sync to servers model is eventually consistent. It will not return errors when an action is performed but rather retry periodically :
* A service register call will always succeed even if the sync fails (we saw that with ACLs pre 0.8 : https://github.com/hashicorp/consul/pull/4771 and agent wipe : https://github.com/hashicorp/consul/pull/5217)
* A node id conflict can prevent a node from registering itself in the catalog

### Issue
Whether the sync failing is due to a bug, a bad setup, or temporary failure it is hard do detect these desyncs at scale :
* When such issues occur the only traces are in the logs
* There is no way to have an overview of an agent sync state

We would like to detect these issues to :
* Monitor the sync status of agents and their services
* Fail early in the case of conflicts

### Proposal
* Add an endpoint such as `/v1/agent/status` returning the state of the agent sync for its node, service , checks. Additionally other things such as the proposed async persistent state (https://github.com/hashicorp/consul/pull/6101) could find there place there too.
* Add metrics for these async elements to enable global monitoring

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the proposed /v1/agent/status endpoint and the existing agent sync behavior described in the issue, including its log traces and service, node, and check state. Done would require an agreed design for reporting sync status and metrics that supports monitoring agents and detecting conflicts; no files or tests are named.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.