cloudnative-pg / cloudnative-pg/grafana-dashboards

Panels labelled {{pod}} draw duplicate series after a pod is recreated

Open
#71 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Makefile
Stars
88
Forks
30
PR merge metrics
No merged PRs in 30d

Description

## What

Around 19 panels plot a metric raw while labelling the series `{{pod}}` — `Blocked Queries`, `Block I/O`, `Temp Bytes`, `Deadlocks`, `WAL Segment Archive Status`, `Archiver Status`, `Last Archive Age`, `WAL Count`, `Errors`, `First Recoverability Point`, `Requested/Timed`, `Write/Sync time`, `Replication Lag`, `Write Lag`, `Flush Lag`, `Replay Lag`, `Version`, and the stat tiles that repeat over `$instances`.

The exporter's series identity includes `instance`, which is the pod's IP and port. When a pod is recreated — failover, rolling upgrade, node replacement — it starts a **new** series for the same pod. Any window spanning that event renders two series with the same legend, one ending where the other begins. On the repeated stat tiles it shows as two numbers inside one tile.

## Evidence

Across a fleet of CNPG instances, over 14 days, 21 of 47 pods carried more than one `instance` label; one carried six. Queried over ten days against one three-instance cluster:

```
cnpg_pg_replication_lag{...} -> 6 series for 3 pods
max by (pod) (cnpg_pg_replication_lag{...}) -> 3 series for the same 3 pods
```

Other metrics on the same cluster, same window:

```
cnpg_collector_postgres_version 8 series -> 4
cnpg_collector_pg_wal 62 series -> 4
cnpg_pg_stat_archiver_archived_count 6 series -> 3
```

## Possible fix

Aggregate each panel by exactly the labels its legend prints — `max by (pod)` for gauges and raw counters, `sum by (pod)` for counter rates so per-device series still add up, and `max by (pod, application_name)` where the legend names the standby. `max` over a single series is that series, so steady state is unchanged and only the duplicates collapse.

I have this implemented and running against production clusters, and I am happy to open a PR. Since it touches ~19 panels, the AI policy's "design first" guidance says to ask before sending something that wide — so: would you like it as one PR, split by row, or not at all?

## Disclosure

Per the project's AI policy: the analysis and the proposed expressions were produced with AI assistance and verified by me against live clusters, with the measurements above.

Contributor guide

Open the contributing guide

Research direction

Locate the dashboard definitions and panel queries for the named metrics, beginning with Replication Lag and the repeated stat tiles. Check each query against the labels shown in its legend and validate the updated panels over a pod recreation window. Done means duplicate series collapse per pod while steady-state values and counter-rate totals remain correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.