Best way to monitor cluster leader presence/absence
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
Our monitoring for cluster leader is based on consul_raft_leader exported by the prometheus consul exporter
https://github.com/prometheus/consul_exporter
As I understand it gets the leader using /v1/status/leader endpoint
However there were many instances where queries to consul (v1/catalog/nodes or v1/catalog/services) were failing with HTTP 500 / "No cluster leader" despite /v1/status/leader saying there was a leader according to each individual node.
Is consul_raft_leader or /v1/status/leader still a good way to monitor for leader presence/absence?
Contributor guide
Research direction
Start by comparing the consul_raft_leader metric from the Prometheus consul exporter with Consul's /v1/status/leader, /v1/catalog/nodes, and /v1/catalog/services endpoints. Reproduce the reported HTTP 500 and "No cluster leader" behavior, then determine whether the monitoring signal reliably reflects catalog availability and document the result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus
- Domain
- api, distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100