cloudnative-pg / cloudnative-pg/grafana-dashboards
Panels labelled {{pod}} draw duplicate series after a pod is recreated
- Dominant language
- Makefile
- Stars
- 88
- Forks
- 30
- PR merge metrics
- No merged PRs in 30d
Description
## What
Around 19 panels plot a metric raw while labelling the series `{{pod}}` — `Blocked Queries`, `Block I/O`, `Temp Bytes`, `Deadlocks`, `WAL Segment Archive Status`, `Archiver Status`, `Last Archive Age`, `WAL Count`, `Errors`, `First Recoverability Point`, `Requested/Timed`, `Write/Sync time`, `Replication Lag`, `Write Lag`, `Flush Lag`, `Replay Lag`, `Version`, and the stat tiles that repeat over `$instances`.
The exporter's series identity includes `instance`, which is the pod's IP and port. When a pod is recreated — failover, rolling upgrade, node replacement — it starts a **new** series for the same pod. Any window spanning that event renders two series with the same legend, one ending where the other begins. On the repeated stat tiles it shows as two numbers inside one tile.
## Evidence
Across a fleet of CNPG instances, over 14 days, 21 of 47 pods carried more than one `instance` label; one carried six. Queried over ten days against one three-instance cluster:
```
cnpg_pg_replication_lag{...} -> 6 series for 3 pods
max by (pod) (cnpg_pg_replication_lag{...}) -> 3 series for the same 3 pods
```
Other metrics on the same cluster, same window:
```
cnpg_collector_postgres_version 8 series -> 4
cnpg_collector_pg_wal 62 series -> 4
cnpg_pg_stat_archiver_archived_count 6 series -> 3
```
## Possible fix
Aggregate each panel by exactly the labels its legend prints — `max by (pod)` for gauges and raw counters, `sum by (pod)` for counter rates so per-device series still add up, and `max by (pod, application_name)` where the legend names the standby. `max` over a single series is that series, so steady state is unchanged and only the duplicates collapse.
I have this implemented and running against production clusters, and I am happy to open a PR. Since it touches ~19 panels, the AI policy's "design first" guidance says to ask before sending something that wide — so: would you like it as one PR, split by row, or not at all?
## Disclosure
Per the project's AI policy: the analysis and the proposed expressions were produced with AI assistance and verified by me against live clusters, with the measurements above.
Contributor guide
Research direction
Locate the dashboard definitions and panel queries for the named metrics, beginning with Replication Lag and the repeated stat tiles. Check each query against the labels shown in its legend and validate the updated panels over a pod recreation window. Done means duplicate series collapse per pod while steady-state values and counter-rate totals remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100