cloudnative-pg / cloudnative-pg/grafana-dashboards

Cluster replication health does not take remote replica clusters into account

Open
#58 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Makefile
Stars
88
Forks
30
PR merge metrics
No merged PRs in 30d

Description

I have a database with 2 remote replica clusters (distributed topology) which are causing the grafana dashboard to report the Cluster Replication Health as `Degraded`.

- cnpg_pg_replication_streaming_replicas = 4, 2 remote and 2 local.

- sum(cnpg_pg_replication_is_wal_receiver_up) = 2 since it can't see the replicas in the remote clusters

Expression used:

```

(

max(cnpg_pg_replication_streaming_replicas{})

-

sum(cnpg_pg_replication_is_wal_receiver_up{})

)

+

(

clamp_max(max(cnpg_pg_replication_streaming_replicas{}), 1)

-

1

)

```

The formula evaluates to (4 - 2) + (1 - 1) = 2 and reports a Degraded cluster replication health, even though there are actually 4 streaming replicas and 4 wal receivers up.

Contributor guide

Open the contributing guide

Research direction

Start in the Grafana dashboard panel that reports Cluster Replication Health and inspect the provided expression using cnpg_pg_replication_streaming_replicas and cnpg_pg_replication_is_wal_receiver_up. Reproduce the calculation with two remote and two local replicas, then verify that the health result reflects all four streaming replicas and WAL receivers rather than reporting Degraded.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.