canonical / canonical/postgresql-operator
Patroni DCS issues on leader when a secondary becomes unreachable
- Dominant language
- Python
- Stars
- 20
- Forks
- 36
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 30
Description
## Steps to reproduce
I have a three node PostgreSQL cluster, using charm revision 936 from channel 14/stable. The backing cloud is OpenStack.
```
dbaas-jaas-ps7@is-bastion-ps7:~$ juju status postgresql
Model Controller Cloud/Region Version SLA Timestamp
dbaas-jaas-ps7 juju-controller-36-cloud-infrastructure-ps7 prodstack7/prodstack7 3.6.12 unsupported 00:01:32Z
App Version Status Scale Charm Channel Rev Exposed Message
aproxy active 3 aproxy latest/edge 19 no Service ready on target proxy egress.ps7.internal:3128
landscape-client active 3 landscape-client latest/stable 72 no Client registered!
postgresql 14.19 active 3 postgresql 14/stable 936 yes
telegraf active 3 telegraf latest/stable 75 yes Monitoring data-integrator/0 (source version/commit 23.10)
ubuntu-advantage active 3 ubuntu-advantage latest/stable 137 no Attached (esm-apps,esm-infra,livepatch)
Unit Workload Agent Machine Public address Ports Message
postgresql/0 active idle 2 10.156.64.29 5432/tcp
aproxy/0 active idle 10.156.64.29 Service ready on target proxy egress.ps7.internal:3128
landscape-client/4 active idle 10.156.64.29 Client registered!
telegraf/4 active idle 10.156.64.29 9103/tcp Monitoring postgresql/0 (source version/commit 23.10)
ubuntu-advantage/4* active idle 10.156.64.29 Attached (esm-apps,esm-infra,livepatch)
postgresql/2* active idle 4 10.156.64.14 5432/tcp Primary
aproxy/4 active idle 10.156.64.14 Service ready on target proxy egress.ps7.internal:3128
landscape-client/3 active idle 10.156.64.14 Client registered!
telegraf/3* active idle 10.156.64.14 9103/tcp Monitoring postgresql/2 (source version/commit 23.10)
ubuntu-advantage/3 active idle 10.156.64.14 Attached (esm-apps,esm-infra,livepatch)
postgresql/3 active idle 7 10.156.64.15 5432/tcp
aproxy/3 active idle 10.156.64.15 Service ready on target proxy egress.ps7.internal:3128
landscape-client/7 active idle 10.156.64.15 Client config updated!
telegraf/7 active idle 10.156.64.15 9103/tcp Monitoring postgresql/3 (source version/commit 23.10)
ubuntu-advantage/7 active idle 10.156.64.15 Attached (esm-apps,esm-infra,livepatch)
Machine State Address Inst id Base AZ Message
2 started 10.156.64.29 47b3f438-09e2-49ce-bf65-5e2c76d4bc23 ubuntu@22.04 AZ2 ACTIVE
4 started 10.156.64.14 e5c31099-91d1-4a08-b4fe-29180fe7322a ubuntu@22.04 AZ1 ACTIVE
7 started 10.156.64.15 d7f96f1e-d156-42a4-a9bb-1d8815d8c8c6 ubuntu@22.04 AZ3 ACTIVE
Offer Application Charm Rev Connected Endpoint Interface Role
postgresql postgresql postgresql 936 2/2 database postgresql_client provider
```
1. Simulate the loss of one cluster node: `openstack server pause 47b3f438-09e2-49ce-bf65-5e2c76d4bc23`
## Expected behavior
Cluster continues to work
## Actual behavior
Cluster goes into read-only mode, with Patroni throwing errors because it can't reach the DCS.
## Versions
Operating system:
Juju CLI: 3.6.12
Juju agent: 3.6.12
Charm revision: 936 (14/stable)
LXD: n/a
## Log output
Juju debug log:
## Additional context
Raft config per-node:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-2:~$ sudo cat /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml | yq '.raft'
data_dir: /var/snap/charmed-postgresql/current/etc/patroni/raft
self_addr: '10.156.64.29:2222'
password:
partner_addrs:
- 10.156.64.15:2222
- 10.156.64.14:2222
---
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo cat /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml | yq '.raft'
data_dir: /var/snap/charmed-postgresql/current/etc/patroni/raft
self_addr: '10.156.64.14:2222'
password:
partner_addrs:
- 10.156.64.29:2222
- 10.156.64.15:2222
---
ubuntu@juju-924e4c-dbaas-jaas-ps7-7:~$ sudo cat /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml | yq '.raft'
data_dir: /var/snap/charmed-postgresql/current/etc/patroni/raft
self_addr: '10.156.64.15:2222'
password:
partner_addrs:
- 10.156.64.29:2222
- 10.156.64.14:2222
```
Raft status:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo -E -H -u snap_daemon charmed-postgresql.syncobj-admin -conn 127.0.0.1:2222 -status -pass
commit_idx: 3146230
enabled_code_version: 0
has_quorum: True
last_applied: 3146230
leader: 10.156.64.14:2222
leader_commit_idx: 3146230
log_len: 12
match_idx_count: 3
match_idx_server_10.156.64.15:2222: 3146230
match_idx_server_10.156.64.29:2222: 3146230
match_idx_server_10.156.64.8:2222: 0
next_node_idx_count: 3
next_node_idx_server_10.156.64.15:2222: 3146231
next_node_idx_server_10.156.64.29:2222: 3146231
next_node_idx_server_10.156.64.8:2222: 3145356
partner_node_status_server_10.156.64.15:2222: 2
partner_node_status_server_10.156.64.29:2222: 2
partner_node_status_server_10.156.64.8:2222: 0
partner_nodes_count: 3
raft_term: 154699
readonly_nodes_count: 0
revision: deprecated
self: 10.156.64.14:2222
self_code_version: 0
state: 2
uptime: 841706
version: 0.3.12
```
Cluster topology:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo -u snap_daemon patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml list
+ Cluster: postgresql (7559832502133297764) -+-----------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+--------------+--------------+--------------+-----------+----+-----------+
| postgresql-0 | 10.156.64.29 | Sync Standby | streaming | 17 | 0 |
| postgresql-2 | 10.156.64.14 | Leader | running | 17 | |
| postgresql-3 | 10.156.64.15 | Sync Standby | streaming | 17 | 0 |
+--------------+--------------+--------------+-----------+----+-----------+
```
Confirming cluster is not in recovery:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ psql -h 10.156.64.14 -U operator -d postgres -W -c 'SELECT pg_is_in_recovery();'
Password:
pg_is_in_recovery
-------------------
f
(1 row)
```
---
Simulating an outage for `postgresql-0`: `openstack server pause 47b3f438-09e2-49ce-bf65-5e2c76d4bc23`
Watching patroni logs on the leader (`sudo tail -F /var/snap/charmed-postgresql/common/var/log/patroni/patroni.log`), we start to see `Error communicating with DCS`:
```
2026-01-07 00:34:20 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
2026-01-07 00:34:30 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
2026-01-07 00:34:40 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:34:50 UTC [45979]: ERROR: Error communicating with DCS
2026-01-07 00:34:50 UTC [45979]: INFO: demoting self because DCS is not accessible and I was a leader
2026-01-07 00:34:50 UTC [45979]: INFO: Demoting self (offline)
2026-01-07 00:35:32 UTC [45979]: INFO: closed patroni connections to postgres
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has become inaccessible: No such file or directory
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has appeared; following new file
2026-01-07 00:35:32 UTC [45979]: INFO: postmaster pid=540777
2026-01-07 00:35:33 UTC [45979]: INFO: demoted self because DCS is not accessible and I was a leader
2026-01-07 00:35:33 UTC [45979]: WARNING: Loop time exceeded, rescheduling immediately.
2026-01-07 00:35:33 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:35:33 UTC [45979]: INFO: establishing a new patroni heartbeat connection to postgres
2026-01-07 00:35:43 UTC [45979]: ERROR: Error communicating with DCS
2026-01-07 00:35:43 UTC [45979]: INFO: DCS is not accessible
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:35:43 UTC [45979]: WARNING: Loop time exceeded, rescheduling immediately.
2026-01-07 00:35:43 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:35:53 UTC [45979]: ERROR: Error communicating with DCS
2026-01-07 00:35:53 UTC [45979]: INFO: DCS is not accessible
2026-01-07 00:35:53 UTC [45979]: WARNING: Loop time exceeded, rescheduling immediately.
2026-01-07 00:35:53 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:36:03 UTC [45979]: ERROR: Error communicating with DCS
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:36:03 UTC [45979]: INFO: DCS is not accessible
2026-01-07 00:36:03 UTC [45979]: WARNING: Loop time exceeded, rescheduling immediately.
```
Cluster goes into recovery:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ psql -h 10.156.64.14 -U operator -d postgres -W -c 'SELECT pg_is_in_recovery();'
Password:
pg_is_in_recovery
-------------------
t
(1 row)
```
Patroni commands hang waiting for raft:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo -u snap_daemon env PATRONI_LOG_LEVEL=DEBUG patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml list
2026-01-06 15:38:25,592 - DEBUG - Loading configuration from file /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml
2026-01-06 15:38:34,247 - INFO - waiting on raft
2026-01-06 15:38:39,248 - INFO - waiting on raft
2026-01-06 15:38:44,248 - INFO - waiting on raft
```
---
Bringing the down node back: `openstack server unpause 47b3f438-09e2-49ce-bf65-5e2c76d4bc23`
Patroni logs on the leader shows recovery:
```
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:36:42 UTC [45979]: INFO: establishing a new patroni restapi connection to postgres
2026-01-07 00:36:33 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:36:42 UTC [45979]: INFO: promoted self to leader because I had the session lock
2026-01-07 00:36:42 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:36:42 UTC [45979]: INFO: updated leader lock during promote
2026-01-07 00:36:43 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:36:43 UTC [45979]: INFO: Assigning synchronous standby status to ['postgresql-3']
2026-01-07 00:36:45 UTC [45979]: INFO: Synchronous standby status assigned to ['postgresql-3']
2026-01-07 00:36:45 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
2026-01-07 00:36:45 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
2026-01-07 00:36:03 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:36:13 UTC [45979]: ERROR: Error communicating with DCS
2026-01-07 00:36:13 UTC [45979]: INFO: DCS is not accessible
2026-01-07 00:36:13 UTC [45979]: WARNING: Loop time exceeded, rescheduling immediately.
2026-01-07 00:37:07 UTC [45979]: INFO: Changed archive_mode from 'on' to 'True' (restart might be required)
2026-01-07 00:37:07 UTC [45979]: INFO: Changed statement_timeout from '2s' to '0'
2026-01-07 00:37:07 UTC [45979]: INFO: Changed synchronous_commit from 'on' to 'True'
2026-01-07 00:37:07 UTC [45979]: INFO: Reloading PostgreSQL configuration.
2026-01-07 00:37:08 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has become inaccessible: No such file or directory
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has appeared; following new file
2026-01-07 00:37:08 UTC [45979]: INFO: No local configuration items changed.
2026-01-07 00:37:08 UTC [45979]: INFO: Changed archive_mode from 'on' to 'True' (restart might be required)
2026-01-07 00:37:08 UTC [45979]: INFO: Changed statement_timeout from '2s' to '0'
2026-01-07 00:37:08 UTC [45979]: INFO: Changed synchronous_commit from 'on' to 'True'
2026-01-07 00:37:08 UTC [45979]: INFO: Reloading PostgreSQL configuration.
2026-01-07 00:37:09 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:37:11 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
2026-01-07 00:37:12 UTC [45979]: INFO: Changed archive_mode from 'on' to 'True' (restart might be required)
2026-01-07 00:37:12 UTC [45979]: INFO: Changed statement_timeout from '2s' to '0'
2026-01-07 00:37:12 UTC [45979]: INFO: Changed synchronous_commit from 'on' to 'True'
2026-01-07 00:37:12 UTC [45979]: INFO: Reloading PostgreSQL configuration.
2026-01-07 00:37:13 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:37:13 UTC [45979]: INFO: No local configuration items changed.
2026-01-07 00:37:13 UTC [45979]: INFO: Changed archive_mode from 'on' to 'True' (restart might be required)
2026-01-07 00:37:13 UTC [45979]: INFO: Changed statement_timeout from '2s' to '0'
2026-01-07 00:37:13 UTC [45979]: INFO: Changed synchronous_commit from 'on' to 'True'
2026-01-07 00:37:13 UTC [45979]: INFO: Reloading PostgreSQL configuration.
2026-01-07 00:37:14 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
```
The down server is no longer listed in the patroni topology:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo -u snap_daemon patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml topology
+ Cluster: postgresql (7559832502133297764) ---+-----------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+----------------+--------------+--------------+-----------+----+-----------+
| postgresql-2 | 10.156.64.14 | Leader | running | 18 | |
| + postgresql-3 | 10.156.64.15 | Sync Standby | streaming | 18 | 0 |
+----------------+--------------+--------------+-----------+----+-----------+
```
That server seems to come back after ~5-10 minutes:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo -u snap_daemon patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml topology
+ Cluster: postgresql (7559832502133297764) ---+-----------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+----------------+--------------+--------------+-----------+----+-----------+
| postgresql-2 | 10.156.64.14 | Leader | running | 18 | |
| + postgresql-0 | 10.156.64.29 | Sync Standby | streaming | 18 | 0 |
| + postgresql-3 | 10.156.64.15 | Sync Standby | streaming | 18 | 0 |
+----------------+--------------+--------------+-----------+----+-----------+
```
Contributor guide
Research direction
Start by reproducing the three-node outage with the documented OpenStack pause command, then inspect patroni.yaml, the Patroni log at /var/snap/charmed-postgresql/common/var/log/patroni/patroni.log, and the syncobj-admin raft status. Compare behavior before, during, and after the secondary becomes unreachable; done means the leader remains writable and Patroni does not demote it when quorum is still available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100