canonical / canonical/postgresql-operator
Confusing Async-replication behavior when Primary cluster and Strandby cluster names (juju app names) are the same
- Dominant language
- Python
- Stars
- 20
- Forks
- 36
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 30
Description
On a dual-cluster setup with asynchronous replication (2+2), after `promote-to-primary scope=unit`, the secondary is stuck in Replica and doesn't go to Sync Standby.
Summary of the setup:
| Member | IP address | Site | Patroni Status |
| ----------- | ----------- | ----------- | ----------- |
| postgresql-0 | 10.10.128.24 | Primary | Sync Standby (primary cluster) |
| postgresql-1 | 10.10.128.23 | Primary | Leader (primary cluster) |
| postgresql-0 | 10.10.118.24 | Secondary | Standby Leader (secondary cluster) |
| postgresql-1 | 10.10.118.23 | Secondary | Replica (secondary cluster) |
## Steps to reproduce
1. On primary cluster, perform a `promote-to-primary` action to switch secondary to primary unit
```
juju run postgresql/0 -- promote-to-primary scope=unit
```
2. Observe patronictl output
## Expected behavior
```
sudo patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml list
+ Cluster: postgresql (7633847608209757521) -+-----------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+--------------+--------------+--------------+-----------+----+-----------+
| postgresql-0 | 10.52.128.24 | Leader | running | 4 | |
| postgresql-1 | 10.52.128.23 | Sync Standby | streaming | 4 | 0 |
+--------------+--------------+--------------+-----------+----+-----------+
```
## Actual behavior
```
sudo patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml list
+ Cluster: postgresql (7633847608209757521) --------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+--------------+--------------+---------+-----------+----+-----------+
| postgresql-0 | 10.52.128.24 | Leader | running | 4 | |
| postgresql-1 | 10.52.128.23 | Replica | streaming | 4 | 0 |
+--------------+--------------+---------+-----------+----+-----------+
```
## Versions
Operating system: Ubuntu 24.04
Juju CLI: 3.6.14
Juju agent: 3.6.14
Charm revision: 1047
LXD: 5.21/stable
## Log output
Juju debug log:
## Additional context
When the promote happens, we now have 2 nodes `postgresql-0` the pg_stat_replication table. First one in `sync` is the primary unit of the secondary cluster. Second one is the replica unit of the main cluster (the one we aimed to failover with the `promote-to-primary` action)
```
postgres=# SELECT application_name, client_addr, state, sync_state FROM pg_stat_replication;
application_name | client_addr | state | sync_state
------------------+--------------+-----------+------------
postgresql-0 | 10.10.118.24 | streaming | sync
postgresql-0 | 10.10.128.24 | streaming | potential
```
Workaround:
- Adjust `synchronous_node_count` from `1` to `2`. This cannot be increased to more than 1 using `juju config`.
```
sudo charmed-postgresql.patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml edit-config
```
- Restart patroni on the main unit of the secondary cluster (10.10.118.24)
```
sudo snap start charmed-postgresql.patroni
```
Now the correct unit (10.10.128.24, new Leader) took over the `sync` state
```
postgres=# SELECT application_name, client_addr, state, sync_state FROM pg_stat_replication;
application_name | client_addr | state | sync_state
------------------+--------------+-----------+------------
postgresql-0 | 10.10.128.24 | streaming | sync
postgresql-1 | 10.10.118.23 | streaming | async
(2 rows)
```
And patronictl returns proper config
```
sudo patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml list
+ Cluster: postgresql (7633847608209757521) -+-----------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+--------------+--------------+--------------+-----------+----+-----------+
| postgresql-0 | 10.10.128.24 | Leader | running | 4 | |
| postgresql-1 | 10.10.128.23 | Sync Standby | streaming | 4 | 0 |
+--------------+--------------+--------------+-----------+----+-----------+
```
Contributor guide
Research direction
Start with the promote-to-primary action and the Patroni configuration used by patronictl, then compare the primary and secondary cluster identities when their Juju application names match. Use pg_stat_replication and the reproduced patronictl output to trace which unit is selected for sync state. Done means the promoted unit becomes Leader and the other primary-cluster unit becomes Sync Standby without the workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100