cloudnative-pg / cloudnative-pg/grafana-dashboards
Cluster replication health does not take remote replica clusters into account
- Dominant language
- Makefile
- Stars
- 88
- Forks
- 30
- PR merge metrics
- No merged PRs in 30d
Description
I have a database with 2 remote replica clusters (distributed topology) which are causing the grafana dashboard to report the Cluster Replication Health as `Degraded`.
- cnpg_pg_replication_streaming_replicas = 4, 2 remote and 2 local.
- sum(cnpg_pg_replication_is_wal_receiver_up) = 2 since it can't see the replicas in the remote clusters
Expression used:
```
(
max(cnpg_pg_replication_streaming_replicas{})
-
sum(cnpg_pg_replication_is_wal_receiver_up{})
)
+
(
clamp_max(max(cnpg_pg_replication_streaming_replicas{}), 1)
-
1
)
```
The formula evaluates to (4 - 2) + (1 - 1) = 2 and reports a Degraded cluster replication health, even though there are actually 4 streaming replicas and 4 wal receivers up.
Contributor guide
Research direction
Start in the Grafana dashboard panel that reports Cluster Replication Health and inspect the provided expression using cnpg_pg_replication_streaming_replicas and cnpg_pg_replication_is_wal_receiver_up. Reproduce the calculation with two remote and two local replicas, then verify that the health result reflects all four streaming replicas and WAL receivers rather than reporting Degraded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100