hashicorp / hashicorp/consul

Best way to monitor cluster leader presence/absence

Open
#10,733 8 comments 0 reactions 0 assignees View on GitHub
theme/telemetry type/question waiting-reply
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

Our monitoring for cluster leader is based on consul_raft_leader exported by the prometheus consul exporter

https://github.com/prometheus/consul_exporter

As I understand it gets the leader using /v1/status/leader endpoint

However there were many instances where queries to consul (v1/catalog/nodes or v1/catalog/services) were failing with HTTP 500 / "No cluster leader" despite /v1/status/leader saying there was a leader according to each individual node.

Is consul_raft_leader or /v1/status/leader still a good way to monitor for leader presence/absence?

Contributor guide

Open the contributing guide

Research direction

Start by comparing the consul_raft_leader metric from the Prometheus consul exporter with Consul's /v1/status/leader, /v1/catalog/nodes, and /v1/catalog/services endpoints. Reproduce the reported HTTP 500 and "No cluster leader" behavior, then determine whether the monitoring signal reliably reflects catalog availability and document the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus
Domain
api, distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.