canonical / canonical/postgresql-operator

Patroni DCS issues on leader when a secondary becomes unreachable

Open
#1,358 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

## Steps to reproduce
I have a three node PostgreSQL cluster, using charm revision 936 from channel 14/stable. The backing cloud is OpenStack.
```
dbaas-jaas-ps7@is-bastion-ps7:~$ juju status postgresql
Model Controller Cloud/Region Version SLA Timestamp
dbaas-jaas-ps7 juju-controller-36-cloud-infrastructure-ps7 prodstack7/prodstack7 3.6.12 unsupported 00:01:32Z

App Version Status Scale Charm Channel Rev Exposed Message
aproxy active 3 aproxy latest/edge 19 no Service ready on target proxy egress.ps7.internal:3128
landscape-client active 3 landscape-client latest/stable 72 no Client registered!
postgresql 14.19 active 3 postgresql 14/stable 936 yes
telegraf active 3 telegraf latest/stable 75 yes Monitoring data-integrator/0 (source version/commit 23.10)
ubuntu-advantage active 3 ubuntu-advantage latest/stable 137 no Attached (esm-apps,esm-infra,livepatch)

Unit Workload Agent Machine Public address Ports Message
postgresql/0 active idle 2 10.156.64.29 5432/tcp
aproxy/0 active idle 10.156.64.29 Service ready on target proxy egress.ps7.internal:3128
landscape-client/4 active idle 10.156.64.29 Client registered!
telegraf/4 active idle 10.156.64.29 9103/tcp Monitoring postgresql/0 (source version/commit 23.10)
ubuntu-advantage/4* active idle 10.156.64.29 Attached (esm-apps,esm-infra,livepatch)
postgresql/2* active idle 4 10.156.64.14 5432/tcp Primary
aproxy/4 active idle 10.156.64.14 Service ready on target proxy egress.ps7.internal:3128
landscape-client/3 active idle 10.156.64.14 Client registered!
telegraf/3* active idle 10.156.64.14 9103/tcp Monitoring postgresql/2 (source version/commit 23.10)
ubuntu-advantage/3 active idle 10.156.64.14 Attached (esm-apps,esm-infra,livepatch)
postgresql/3 active idle 7 10.156.64.15 5432/tcp
aproxy/3 active idle 10.156.64.15 Service ready on target proxy egress.ps7.internal:3128
landscape-client/7 active idle 10.156.64.15 Client config updated!
telegraf/7 active idle 10.156.64.15 9103/tcp Monitoring postgresql/3 (source version/commit 23.10)
ubuntu-advantage/7 active idle 10.156.64.15 Attached (esm-apps,esm-infra,livepatch)

Machine State Address Inst id Base AZ Message
2 started 10.156.64.29 47b3f438-09e2-49ce-bf65-5e2c76d4bc23 ubuntu@22.04 AZ2 ACTIVE
4 started 10.156.64.14 e5c31099-91d1-4a08-b4fe-29180fe7322a ubuntu@22.04 AZ1 ACTIVE
7 started 10.156.64.15 d7f96f1e-d156-42a4-a9bb-1d8815d8c8c6 ubuntu@22.04 AZ3 ACTIVE

Offer Application Charm Rev Connected Endpoint Interface Role
postgresql postgresql postgresql 936 2/2 database postgresql_client provider
```

1. Simulate the loss of one cluster node: `openstack server pause 47b3f438-09e2-49ce-bf65-5e2c76d4bc23`

## Expected behavior
Cluster continues to work

## Actual behavior
Cluster goes into read-only mode, with Patroni throwing errors because it can't reach the DCS.

## Versions

Operating system:

Juju CLI: 3.6.12

Juju agent: 3.6.12

Charm revision: 936 (14/stable)

LXD: n/a

## Log output

Juju debug log:

## Additional context

Raft config per-node:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-2:~$ sudo cat /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml | yq '.raft'
data_dir: /var/snap/charmed-postgresql/current/etc/patroni/raft
self_addr: '10.156.64.29:2222'
password:
partner_addrs:
- 10.156.64.15:2222
- 10.156.64.14:2222

---

ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo cat /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml | yq '.raft'
data_dir: /var/snap/charmed-postgresql/current/etc/patroni/raft
self_addr: '10.156.64.14:2222'
password:
partner_addrs:
- 10.156.64.29:2222
- 10.156.64.15:2222

---

ubuntu@juju-924e4c-dbaas-jaas-ps7-7:~$ sudo cat /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml | yq '.raft'
data_dir: /var/snap/charmed-postgresql/current/etc/patroni/raft
self_addr: '10.156.64.15:2222'
password:
partner_addrs:
- 10.156.64.29:2222
- 10.156.64.14:2222
```

Raft status:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo -E -H -u snap_daemon charmed-postgresql.syncobj-admin -conn 127.0.0.1:2222 -status -pass
commit_idx: 3146230
enabled_code_version: 0
has_quorum: True
last_applied: 3146230
leader: 10.156.64.14:2222
leader_commit_idx: 3146230
log_len: 12
match_idx_count: 3
match_idx_server_10.156.64.15:2222: 3146230
match_idx_server_10.156.64.29:2222: 3146230
match_idx_server_10.156.64.8:2222: 0
next_node_idx_count: 3
next_node_idx_server_10.156.64.15:2222: 3146231
next_node_idx_server_10.156.64.29:2222: 3146231
next_node_idx_server_10.156.64.8:2222: 3145356
partner_node_status_server_10.156.64.15:2222: 2
partner_node_status_server_10.156.64.29:2222: 2
partner_node_status_server_10.156.64.8:2222: 0
partner_nodes_count: 3
raft_term: 154699
readonly_nodes_count: 0
revision: deprecated
self: 10.156.64.14:2222
self_code_version: 0
state: 2
uptime: 841706
version: 0.3.12
```

Cluster topology:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo -u snap_daemon patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml list
+ Cluster: postgresql (7559832502133297764) -+-----------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+--------------+--------------+--------------+-----------+----+-----------+
| postgresql-0 | 10.156.64.29 | Sync Standby | streaming | 17 | 0 |
| postgresql-2 | 10.156.64.14 | Leader | running | 17 | |
| postgresql-3 | 10.156.64.15 | Sync Standby | streaming | 17 | 0 |
+--------------+--------------+--------------+-----------+----+-----------+
```
Confirming cluster is not in recovery:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ psql -h 10.156.64.14 -U operator -d postgres -W -c 'SELECT pg_is_in_recovery();'
Password:
pg_is_in_recovery
-------------------
f
(1 row)
```

---

Simulating an outage for `postgresql-0`: `openstack server pause 47b3f438-09e2-49ce-bf65-5e2c76d4bc23`

Watching patroni logs on the leader (`sudo tail -F /var/snap/charmed-postgresql/common/var/log/patroni/patroni.log`), we start to see `Error communicating with DCS`:
```
2026-01-07 00:34:20 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
2026-01-07 00:34:30 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
2026-01-07 00:34:40 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:34:50 UTC [45979]: ERROR: Error communicating with DCS
2026-01-07 00:34:50 UTC [45979]: INFO: demoting self because DCS is not accessible and I was a leader
2026-01-07 00:34:50 UTC [45979]: INFO: Demoting self (offline)
2026-01-07 00:35:32 UTC [45979]: INFO: closed patroni connections to postgres
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has become inaccessible: No such file or directory
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has appeared; following new file
2026-01-07 00:35:32 UTC [45979]: INFO: postmaster pid=540777
2026-01-07 00:35:33 UTC [45979]: INFO: demoted self because DCS is not accessible and I was a leader
2026-01-07 00:35:33 UTC [45979]: WARNING: Loop time exceeded, rescheduling immediately.
2026-01-07 00:35:33 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:35:33 UTC [45979]: INFO: establishing a new patroni heartbeat connection to postgres
2026-01-07 00:35:43 UTC [45979]: ERROR: Error communicating with DCS
2026-01-07 00:35:43 UTC [45979]: INFO: DCS is not accessible
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:35:43 UTC [45979]: WARNING: Loop time exceeded, rescheduling immediately.
2026-01-07 00:35:43 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:35:53 UTC [45979]: ERROR: Error communicating with DCS
2026-01-07 00:35:53 UTC [45979]: INFO: DCS is not accessible
2026-01-07 00:35:53 UTC [45979]: WARNING: Loop time exceeded, rescheduling immediately.
2026-01-07 00:35:53 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:36:03 UTC [45979]: ERROR: Error communicating with DCS
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:36:03 UTC [45979]: INFO: DCS is not accessible
2026-01-07 00:36:03 UTC [45979]: WARNING: Loop time exceeded, rescheduling immediately.
```

Cluster goes into recovery:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ psql -h 10.156.64.14 -U operator -d postgres -W -c 'SELECT pg_is_in_recovery();'
Password:
pg_is_in_recovery
-------------------
t
(1 row)
```

Patroni commands hang waiting for raft:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo -u snap_daemon env PATRONI_LOG_LEVEL=DEBUG patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml list
2026-01-06 15:38:25,592 - DEBUG - Loading configuration from file /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml
2026-01-06 15:38:34,247 - INFO - waiting on raft
2026-01-06 15:38:39,248 - INFO - waiting on raft
2026-01-06 15:38:44,248 - INFO - waiting on raft
```

---

Bringing the down node back: `openstack server unpause 47b3f438-09e2-49ce-bf65-5e2c76d4bc23`

Patroni logs on the leader shows recovery:
```
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:36:42 UTC [45979]: INFO: establishing a new patroni restapi connection to postgres
2026-01-07 00:36:33 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:36:42 UTC [45979]: INFO: promoted self to leader because I had the session lock
2026-01-07 00:36:42 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:36:42 UTC [45979]: INFO: updated leader lock during promote
2026-01-07 00:36:43 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:36:43 UTC [45979]: INFO: Assigning synchronous standby status to ['postgresql-3']
2026-01-07 00:36:45 UTC [45979]: INFO: Synchronous standby status assigned to ['postgresql-3']
2026-01-07 00:36:45 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
2026-01-07 00:36:45 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
2026-01-07 00:36:03 UTC [45979]: INFO: Lock owner: postgresql-2; I am postgresql-2
2026-01-07 00:36:13 UTC [45979]: ERROR: Error communicating with DCS
2026-01-07 00:36:13 UTC [45979]: INFO: DCS is not accessible
2026-01-07 00:36:13 UTC [45979]: WARNING: Loop time exceeded, rescheduling immediately.
2026-01-07 00:37:07 UTC [45979]: INFO: Changed archive_mode from 'on' to 'True' (restart might be required)
2026-01-07 00:37:07 UTC [45979]: INFO: Changed statement_timeout from '2s' to '0'
2026-01-07 00:37:07 UTC [45979]: INFO: Changed synchronous_commit from 'on' to 'True'
2026-01-07 00:37:07 UTC [45979]: INFO: Reloading PostgreSQL configuration.
2026-01-07 00:37:08 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has become inaccessible: No such file or directory
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has appeared; following new file
2026-01-07 00:37:08 UTC [45979]: INFO: No local configuration items changed.
2026-01-07 00:37:08 UTC [45979]: INFO: Changed archive_mode from 'on' to 'True' (restart might be required)
2026-01-07 00:37:08 UTC [45979]: INFO: Changed statement_timeout from '2s' to '0'
2026-01-07 00:37:08 UTC [45979]: INFO: Changed synchronous_commit from 'on' to 'True'
2026-01-07 00:37:08 UTC [45979]: INFO: Reloading PostgreSQL configuration.
2026-01-07 00:37:09 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:37:11 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
2026-01-07 00:37:12 UTC [45979]: INFO: Changed archive_mode from 'on' to 'True' (restart might be required)
2026-01-07 00:37:12 UTC [45979]: INFO: Changed statement_timeout from '2s' to '0'
2026-01-07 00:37:12 UTC [45979]: INFO: Changed synchronous_commit from 'on' to 'True'
2026-01-07 00:37:12 UTC [45979]: INFO: Reloading PostgreSQL configuration.
2026-01-07 00:37:13 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
tail: '/var/snap/charmed-postgresql/common/var/log/patroni/patroni.log' has been replaced; following new file
2026-01-07 00:37:13 UTC [45979]: INFO: No local configuration items changed.
2026-01-07 00:37:13 UTC [45979]: INFO: Changed archive_mode from 'on' to 'True' (restart might be required)
2026-01-07 00:37:13 UTC [45979]: INFO: Changed statement_timeout from '2s' to '0'
2026-01-07 00:37:13 UTC [45979]: INFO: Changed synchronous_commit from 'on' to 'True'
2026-01-07 00:37:13 UTC [45979]: INFO: Reloading PostgreSQL configuration.
2026-01-07 00:37:14 UTC [45979]: INFO: no action. I am (postgresql-2), the leader with the lock
```

The down server is no longer listed in the patroni topology:
```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo -u snap_daemon patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml topology
+ Cluster: postgresql (7559832502133297764) ---+-----------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+----------------+--------------+--------------+-----------+----+-----------+
| postgresql-2 | 10.156.64.14 | Leader | running | 18 | |
| + postgresql-3 | 10.156.64.15 | Sync Standby | streaming | 18 | 0 |
+----------------+--------------+--------------+-----------+----+-----------+
```

That server seems to come back after ~5-10 minutes:

```
ubuntu@juju-924e4c-dbaas-jaas-ps7-4:~$ sudo -u snap_daemon patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml topology
+ Cluster: postgresql (7559832502133297764) ---+-----------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+----------------+--------------+--------------+-----------+----+-----------+
| postgresql-2 | 10.156.64.14 | Leader | running | 18 | |
| + postgresql-0 | 10.156.64.29 | Sync Standby | streaming | 18 | 0 |
| + postgresql-3 | 10.156.64.15 | Sync Standby | streaming | 18 | 0 |
+----------------+--------------+--------------+-----------+----+-----------+
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the three-node outage with the documented OpenStack pause command, then inspect patroni.yaml, the Patroni log at /var/snap/charmed-postgresql/common/var/log/patroni/patroni.log, and the syncobj-admin raft status. Compare behavior before, during, and after the secondary becomes unreachable; done means the leader remains writable and Patroni does not demote it when quorum is still available.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.