canonical / canonical/postgresql-operator
MAAS region DB is read-only because Patroni snap comes up disabled after host reboot, leaving no primary
- Dominant language
- Python
- Stars
- 20
- Forks
- 36
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 30
Description
## Root cause framing (important for triage)
`snap services charmed-postgresql.patroni` showing `Startup=disabled` is *expected* and not the
bug — the snap service is intentionally `disabled` and the charm brings Patroni up via its
`start`/`update-status` hooks (see #1083's `snap services` output:
`charmed-postgresql.patroni disabled active`). This report is therefore **not** requesting that
the snap be enabled-on-boot; it is reporting that the charm's post-reboot reconciliation does not
start Patroni.
This differs from #986, where the `start` hook early-exits because the unit is in `blocked` state
(`Early exit on_start: Unit blocked`). Here the units were `active/idle` at reboot and the failure
still occurs, so #986's blocked-state path does not cover it.
This appears to be a recurrence/regression of the host-reboot fix in #47 (DPE-908), which enabled
Patroni to start on reboot.
## Steps to reproduce
1. Deploy charmed-postgresql (snap) as a 3-node Patroni cluster (VM-hosted).
2. Reboot a host running one of the members (ordinary or ungraceful).
3. Check the snap service state on the rebooted member: `snap services charmed-postgresql.patroni`.
4. If this leaves fewer than 2 members up, no primary is elected.
5. Issue any MAAS-writing Juju operation (e.g. `juju deploy`).
## Expected behavior
The charm's `start`/`update-status` hooks detect that the Patroni snap service is inactive after a reboot and start it, regardless of whether the unit was `active` or `blocked` before the reboot. The cluster then re-elects a primary with no manual `snap start --enable` step.
## Actual behavior
`snap services charmed-postgresql.patroni` shows `disabled` on the rebooted member (and does not auto-start). With >= 2 of 3 members down there is no elected primary, so the surviving member serves read-only and every write fails. Juju surfaces:
```
ERROR ... creating MAAS environ: unexpected: ServerError: 500 Internal Server Error (cannot execute INSERT in a read-only transaction)
```
## Versions
Operating system: Ubuntu 24.04
Juju CLI: 3.6.25
Juju agent: 3.6.25
Charm revision:
LXD: n/a (VM-hosted)
## Log output
Juju debug log:
## Additional context
- The cluster is used as the MAAS 3.7 region DB (`maas_region_db`); regiond `database_host` is
pinned to `postgresql-2`.
- Recurring: observed after single-host reboots and after simultaneous reboots.
- No data loss; the already-running cloud is unaffected. Impact is Medium-High — blocks all
deploy/add-machine/config-write operations.
Workaround (used in production):
```
sudo snap start --enable charmed-postgresql.patroni # on all 3 members
# if the leader is not the member regiond points at:
patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml switchover postgresql --candidate postgresql-2 --force
```
## Requested fix
Reconcile Patroni unconditionally in the `start` hook (not gated on unit status), or enable-on-boot + a post-reboot health check. Either way, a rebooted `active` member must not remain `inactive` and silently serve read-only.
Contributor guide
Research direction
Start with the charm's start and update-status hooks, using the reported distinction between active and blocked units as the key reproduction case. Reboot a member in a three-node cluster and check `snap services charmed-postgresql.patroni`; done means the hook starts inactive Patroni after reboot and the cluster can elect a primary without manual snap commands.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100