canonical / canonical/postgresql-operator

Monitor cross-dc-relation (Feature?)

Open
#1,808 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

When you have cross-site replication, you only notice juju issues when you try to switch the sites.
I had the situation where the cross-site-replication was fine but juju had issues with the cross-site relation. It only showed up when I tried to to the switchover. Which is usually a bit too late.

## Steps to reproduce
1. Deploy two postgresql
2. setup cross site replication
3. have a juju issue (e.g. user accidentally disabled)

## Expected behavior
The charm realizes that it cannot reach the other juju controller anymore and warns the operator about the issue.

## Actual behavior
Nothing happens, but if you try to switchover the sites, it will not work and the consumer site may go into error states. Once that happens, you also see the CMR being in failed state.

## Versions

Operating system: Ubuntu 24.04.4

Juju CLI: 3.5.7

Juju agent: 3.6.12

Charm revision: 1158

LXD: 5.0.6-7fc3b36

## Log output
```
controller-0: 08:12:58 INFO juju.worker.remoterelations cmr start "hdev-swarm-a-postgresql-replication-offer"
controller-0: 08:12:58 ERROR juju.worker.remoterelations cmr error in remote application worker for hdev-swarm-a-postgresql-replication-offer: watching status for offer: cannot get discharge from "https://10.33.236.139:17070/offeraccess": third party refu
sed discharge: cannot discharge: permission denied
controller-0: 08:12:58 INFO juju.worker.remoterelations cmr stopped "hdev-swarm-a-postgresql-replication-offer", err: watching status for offer: cannot get discharge from "https://10.33.236.139:17070/offeraccess": third party refused discharge: cannot dis
charge: permission denied
controller-0: 08:12:58 INFO juju.worker.remoterelations cmr non-fatal error "hdev-swarm-a-postgresql-replication-offer": watching status for offer: cannot get discharge from "https://10.33.236.139:17070/offeraccess": third party refused discharge: cannot
discharge: permission denied
controller-0: 08:12:58 ERROR juju.worker.remoterelations cmr exited "hdev-swarm-a-postgresql-replication-offer": watching status for offer: cannot get discharge from "https://10.33.236.139:17070/offeraccess": third party refused discharge: cannot discharg
e: permission denied
controller-0: 08:12:58 INFO juju.worker.remoterelations cmr restarting "hdev-swarm-a-postgresql-replication-offer" in 15s
```
Digging in the controller logs reveals:
```
controller-1: 07:52:34 ERROR juju.worker.remoterelations cmr error in remote application worker for hdev-swarm-b-postgresql-replication-offer: watching status for offer: cannot get discharge from "https://10.33.236.76:17070/offeraccess": third party refused discharge: cannot discharge: permission denied
```

## Additional context

Contributor guide

Open the contributing guide

Research direction

Reproduce the two-site PostgreSQL cross-site replication setup and the disabled-user or permission failure described in the steps. Start with the Juju remote-relations worker and the controller logs showing the CMR discharge error. Done means the operator is warned about the broken cross-site relation before a switchover is attempted.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.