canonical / canonical/postgresql-operator

MAAS region DB is read-only because Patroni snap comes up disabled after host reboot, leaving no primary

Open
#1,907 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

## Root cause framing (important for triage)

`snap services charmed-postgresql.patroni` showing `Startup=disabled` is *expected* and not the
bug — the snap service is intentionally `disabled` and the charm brings Patroni up via its
`start`/`update-status` hooks (see #1083's `snap services` output:
`charmed-postgresql.patroni disabled active`). This report is therefore **not** requesting that
the snap be enabled-on-boot; it is reporting that the charm's post-reboot reconciliation does not
start Patroni.

This differs from #986, where the `start` hook early-exits because the unit is in `blocked` state
(`Early exit on_start: Unit blocked`). Here the units were `active/idle` at reboot and the failure
still occurs, so #986's blocked-state path does not cover it.

This appears to be a recurrence/regression of the host-reboot fix in #47 (DPE-908), which enabled
Patroni to start on reboot.

## Steps to reproduce

1. Deploy charmed-postgresql (snap) as a 3-node Patroni cluster (VM-hosted).
2. Reboot a host running one of the members (ordinary or ungraceful).
3. Check the snap service state on the rebooted member: `snap services charmed-postgresql.patroni`.
4. If this leaves fewer than 2 members up, no primary is elected.
5. Issue any MAAS-writing Juju operation (e.g. `juju deploy`).

## Expected behavior

The charm's `start`/`update-status` hooks detect that the Patroni snap service is inactive after a reboot and start it, regardless of whether the unit was `active` or `blocked` before the reboot. The cluster then re-elects a primary with no manual `snap start --enable` step.

## Actual behavior

`snap services charmed-postgresql.patroni` shows `disabled` on the rebooted member (and does not auto-start). With >= 2 of 3 members down there is no elected primary, so the surviving member serves read-only and every write fails. Juju surfaces:

```
ERROR ... creating MAAS environ: unexpected: ServerError: 500 Internal Server Error (cannot execute INSERT in a read-only transaction)
```

## Versions

Operating system: Ubuntu 24.04

Juju CLI: 3.6.25

Juju agent: 3.6.25

Charm revision:

LXD: n/a (VM-hosted)

## Log output

Juju debug log:

## Additional context

- The cluster is used as the MAAS 3.7 region DB (`maas_region_db`); regiond `database_host` is
pinned to `postgresql-2`.
- Recurring: observed after single-host reboots and after simultaneous reboots.
- No data loss; the already-running cloud is unaffected. Impact is Medium-High — blocks all
deploy/add-machine/config-write operations.

Workaround (used in production):

```
sudo snap start --enable charmed-postgresql.patroni # on all 3 members
# if the leader is not the member regiond points at:
patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml switchover postgresql --candidate postgresql-2 --force
```

## Requested fix

Reconcile Patroni unconditionally in the `start` hook (not gated on unit status), or enable-on-boot + a post-reboot health check. Either way, a rebooted `active` member must not remain `inactive` and silently serve read-only.

Contributor guide

Open the contributing guide

Research direction

Start with the charm's start and update-status hooks, using the reported distinction between active and blocked units as the key reproduction case. Reboot a member in a three-node cluster and check `snap services charmed-postgresql.patroni`; done means the hook starts inactive Patroni after reboot and the cluster can elect a primary without manual snap commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.