metrics: add alert for PD etcd-leader node stuck in non-PD-leader state
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
### Problem
When the PD service layer (`service_member_role`) drops from leader (`1`) to non-leader (`0`) and **never recovers** — while the embedded etcd layer stays stable (`etcd_server_is_leader` unchanged) and the process has not restarted — no alert fires.
A full audit of all existing rules confirms the gap:
| Alert | Why it stays silent |
|---|---|
| `PD_leader_lease_drop_without_failover` | requires `service_member_role == 1` at eval time **and** `changes >= 2`; a persistent drop gives `changes == 1` and current value `0` |
| `PD_leader_change` | detects TSO-save handoff between ≥2 instances; with no active PD leader, no TSO saves are emitted |
| `PD_cluster_down_store_nums` / `PD_miss_peer_region_count` / region/store health alerts | rely on `pd_cluster_status` / `pd_regions_status`, emitted only by the active PD leader — which does not exist in this scenario |
| All other rules | unrelated to PD service leadership |
Net effect: the cluster has no PD leader serving requests, the embedded etcd is healthy, the process is running — and the entire alert suite is silent.
### Proposed Solution
Add a new alert that fires when the etcd-leader node's PD service layer is not serving as PD leader for a sustained period:
```promql
(service_member_role{job="pd",service="PD"} == 0)
and on(instance,job) (etcd_server_is_leader{job="pd"} == 1)
```
When a normal failover occurs, the node losing etcd leadership also transitions `etcd_server_is_leader` to `0`, so the join condition naturally goes false and no spurious alert fires. The `for: 1m` duration absorbs brief transients during startup or re-election.
Contributor guide
Assessment
This issue has not been assessed yet.