tikv / tikv/pd

metrics: add alert for PD etcd-leader node stuck in non-PD-leader state

Open
#10,320 1 comment 0 reactions 0 assignees View on GitHub
contribution type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

### Problem

When the PD service layer (`service_member_role`) drops from leader (`1`) to non-leader (`0`) and **never recovers** — while the embedded etcd layer stays stable (`etcd_server_is_leader` unchanged) and the process has not restarted — no alert fires.

A full audit of all existing rules confirms the gap:

| Alert | Why it stays silent |
|---|---|
| `PD_leader_lease_drop_without_failover` | requires `service_member_role == 1` at eval time **and** `changes >= 2`; a persistent drop gives `changes == 1` and current value `0` |
| `PD_leader_change` | detects TSO-save handoff between ≥2 instances; with no active PD leader, no TSO saves are emitted |
| `PD_cluster_down_store_nums` / `PD_miss_peer_region_count` / region/store health alerts | rely on `pd_cluster_status` / `pd_regions_status`, emitted only by the active PD leader — which does not exist in this scenario |
| All other rules | unrelated to PD service leadership |

Net effect: the cluster has no PD leader serving requests, the embedded etcd is healthy, the process is running — and the entire alert suite is silent.

### Proposed Solution

Add a new alert that fires when the etcd-leader node's PD service layer is not serving as PD leader for a sustained period:

```promql
(service_member_role{job="pd",service="PD"} == 0)
and on(instance,job) (etcd_server_is_leader{job="pd"} == 1)
```

When a normal failover occurs, the node losing etcd leadership also transitions `etcd_server_is_leader` to `0`, so the join condition naturally goes false and no spurious alert fires. The `for: 1m` duration absorbs brief transients during startup or re-election.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.