cockroachdb / cockroachdb/cockroach
kvserver: leaderless/leaseless unavailability is invisible in metrics (ranges.unavailable reads 0)
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Problem
`ranges.unavailable` counts only quorum loss: ranges with fewer than a quorum of replicas on live nodes. Ranges that have all replicas on live nodes but **no raft leader and no lease** — which cannot serve any traffic — are not counted, and no other metric counts them either.
In a recent escalation on a large multi-tenant cluster, roughly half of ~180k ranges were leaderless and leaseless for ~7 hours (mass election failure under CPU/transport overload, with leader leases enabled). Throughout the outage `ranges.unavailable` read 0 and the DB Console's "Unavailable Ranges" panel showed green, while client traffic to those ranges failed. Responders had to infer the leaderless population indirectly (comparing `replicas.leaders` against range counts).
Notably, the system already recognizes this state internally: the `leaderlessWatcher` (`kv.replica_raft.leaderless_unavailable_threshold`, default 60s) marks replicas that have been leaderless past the threshold as unavailable and fast-fails requests to them with `ReplicaUnavailableError`. During the escalation these circuit breakers were presumably tripped on tens of thousands of replicas — the product was actively telling *clients* the ranges were unavailable while telling *operators* nothing was. The watcher state is not exported as a metric.
Under leader leases this gap is wider than it was historically: availability now requires a completed election plus fortification, so "all replicas on live nodes" is a much weaker proxy for "serving" than it was under epoch leases.
## Proposal
1. **Export the leaderlessWatcher state as a gauge** — e.g. `replicas.leaderless.unavailable` (count of replicas currently tripped). Because the watcher is already duration-thresholded, the gauge is naturally immune to transient election blips and is safe to alert on (healthy value: 0). Optionally also a raw `replicas.leaderless` gauge without the threshold for diagnostics.
2. **Surface it in the DB Console** alongside the existing Unavailable Ranges panel, and mention in that panel's docs that it counts quorum loss only.
3. **Do not fold leaderless into `ranges.unavailable`**: the two conditions require different operator responses (quorum loss = data at risk, cannot self-heal; leaderless = self-healing once capacity/elections recover), and existing dashboards/alerts assume the current semantics.
## Environment
Observed on v26.2.x with leader leases enabled. Related context: #173230 (same escalation).
Jira issue: CRDB-66552
Contributor guide
Research direction
Start by tracing the leaderlessWatcher state and the existing ranges.unavailable metric, then locate the DB Console's Unavailable Ranges panel and its documentation. Done means exporting the thresholded leaderless state, surfacing it alongside the panel, and clarifying that ranges.unavailable counts quorum loss only without changing that metric's semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100