canonical / canonical/postgresql-operator
Fresh deploy / scale-up can render an incomplete raft.partner_addrs mesh, isolating a node from the Raft group
- Dominant language
- Python
- Stars
- 20
- Forks
- 36
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 30
Description
## Bug description
On a fresh multi-unit `postgresql` deployment (and also observed after scaling a unit back up
following a scale-down-to-1), the charm can render an **incomplete/asymmetric `raft.partner_addrs`**
list in `patroni.yaml` on one or more units. Instead of listing all other N-1 peers, an affected
node's `partner_addrs` lists only **one** of the other peers.
This leaves the affected node Raft-isolated: it never actually joins the real Raft group and
instead endlessly tries to become the initial cluster bootstrapper, while the *other* two nodes
can still form a (2-node) Raft group between themselves. Crucially, **PostgreSQL-level streaming
replication between the correctly-paired nodes looks completely healthy**, which makes this bug
easy to misdiagnose as a routine DCS/network blip rather than a charm config-rendering bug,
since the top-level symptom overlaps with several other known issues (e.g. #571, #573).
## Steps observed to reproduce
We hit this twice in slightly different circumstances on Prodstack7/JAAS (3-unit `postgresql`
16/stable rev 1047, Patroni w/ Raft as DCS):
1. Fresh 3-unit deploy (all 3 units created together from scratch).
2. Also observed: scale a 3-unit cluster down to 1 unit (e.g. to perform an S3 backup restore),
then scale back up to 3 by adding 2 units back.
In both cases, one node ended up with a `partner_addrs` list missing at least one peer, e.g.:
```
postgresql-A (10.x.x.195): partner_addrs: [10.x.x.137] # missing peer B (10.x.x.82)
postgresql-B (10.x.x.82): partner_addrs: [10.x.x.137] # missing peer A (10.x.x.195)
postgresql-C (10.x.x.137): partner_addrs: [10.x.x.195] # missing peer B (10.x.x.82)
```
## Symptoms
- `juju status` shows the affected unit stuck (e.g. "awaiting for member to start" indefinitely).
- Patroni log on the affected unit shows `Lock owner: None` / `failed to acquire initialize lock`
in a tight retry loop, instead of joining the existing cluster.
- **Diagnostic tell**: the affected node's on-disk Raft journal file
(`/var/snap/charmed-postgresql/common/data/raft/...`) stays tiny (~1KB), vs. an actively
participating node's journal (~256KB+), because it never actually joins any real Raft group.
- Meanwhile `pg_stat_replication` on the leader and streaming state on the correctly-paired
replica both look completely healthy; there is no indication at the Postgres level that
anything is wrong.
- `patronictl list`/`topology` may hang or return inconsistent results while only some nodes have
been restarted with corrected config (transient, see workaround below).
## Workaround (manual, not a real fix)
On every unit, edit `/var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml` to ensure
`raft.partner_addrs` lists **all** other N-1 peer IPs (not just one). Then restart Patroni
(`sudo snap restart charmed-postgresql.patroni`) **in this order**:
1. The broken/isolated node(s) first (safe, since it has nothing to lose).
2. Other healthy replicas.
3. The current leader last (this causes an expected, brief re-election).
After all N nodes are restarted with the corrected, complete `partner_addrs` list, `patronictl
list` should show all nodes as Leader/Replica with matching timelines and 0 lag within
~30-60 seconds.
**Caveat**: this is a stopgap. Hand-editing `patroni.yaml` may be overwritten by the charm on a
future hook fire (e.g. any subsequent config-changed/relation-changed event that re-renders
Patroni config), so the underlying rendering bug should be fixed at the charm level.
## Environment
- Charm: `postgresql` rev 1047, channel `16/stable`
- Juju via JAAS (Prodstack7 / OpenStack), 3-unit deployment
- Patroni's DCS backend: Raft (not etcd)
## Suggested area to investigate
The charm code path that renders `raft.partner_addrs` into `patroni.yaml` on
scale-up/relation-joined for peer units. Likely a race or stale-relation-data issue where not
all peer IPs are known yet at the time the config is rendered for a given unit, and the charm
doesn't re-render it once the full peer set becomes available.
Contributor guide
Research direction
Trace the charm code path that renders raft.partner_addrs into patroni.yaml during scale-up and peer relation events. Reproduce with a three-unit deployment, inspect each unit's patroni.yaml and Raft journal, and compare juju status with patronictl list. Done means every unit receives all N-1 peer addresses and the full Raft group forms without manual edits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100