canonical / canonical/postgresql-operator

Fresh deploy / scale-up can render an incomplete raft.partner_addrs mesh, isolating a node from the Raft group

Open
#1,910 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

## Bug description

On a fresh multi-unit `postgresql` deployment (and also observed after scaling a unit back up
following a scale-down-to-1), the charm can render an **incomplete/asymmetric `raft.partner_addrs`**
list in `patroni.yaml` on one or more units. Instead of listing all other N-1 peers, an affected
node's `partner_addrs` lists only **one** of the other peers.

This leaves the affected node Raft-isolated: it never actually joins the real Raft group and
instead endlessly tries to become the initial cluster bootstrapper, while the *other* two nodes
can still form a (2-node) Raft group between themselves. Crucially, **PostgreSQL-level streaming
replication between the correctly-paired nodes looks completely healthy**, which makes this bug
easy to misdiagnose as a routine DCS/network blip rather than a charm config-rendering bug,
since the top-level symptom overlaps with several other known issues (e.g. #571, #573).

## Steps observed to reproduce

We hit this twice in slightly different circumstances on Prodstack7/JAAS (3-unit `postgresql`
16/stable rev 1047, Patroni w/ Raft as DCS):

1. Fresh 3-unit deploy (all 3 units created together from scratch).
2. Also observed: scale a 3-unit cluster down to 1 unit (e.g. to perform an S3 backup restore),
then scale back up to 3 by adding 2 units back.

In both cases, one node ended up with a `partner_addrs` list missing at least one peer, e.g.:

```
postgresql-A (10.x.x.195): partner_addrs: [10.x.x.137] # missing peer B (10.x.x.82)
postgresql-B (10.x.x.82): partner_addrs: [10.x.x.137] # missing peer A (10.x.x.195)
postgresql-C (10.x.x.137): partner_addrs: [10.x.x.195] # missing peer B (10.x.x.82)
```

## Symptoms

- `juju status` shows the affected unit stuck (e.g. "awaiting for member to start" indefinitely).
- Patroni log on the affected unit shows `Lock owner: None` / `failed to acquire initialize lock`
in a tight retry loop, instead of joining the existing cluster.
- **Diagnostic tell**: the affected node's on-disk Raft journal file
(`/var/snap/charmed-postgresql/common/data/raft/...`) stays tiny (~1KB), vs. an actively
participating node's journal (~256KB+), because it never actually joins any real Raft group.
- Meanwhile `pg_stat_replication` on the leader and streaming state on the correctly-paired
replica both look completely healthy; there is no indication at the Postgres level that
anything is wrong.
- `patronictl list`/`topology` may hang or return inconsistent results while only some nodes have
been restarted with corrected config (transient, see workaround below).

## Workaround (manual, not a real fix)

On every unit, edit `/var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml` to ensure
`raft.partner_addrs` lists **all** other N-1 peer IPs (not just one). Then restart Patroni
(`sudo snap restart charmed-postgresql.patroni`) **in this order**:

1. The broken/isolated node(s) first (safe, since it has nothing to lose).
2. Other healthy replicas.
3. The current leader last (this causes an expected, brief re-election).

After all N nodes are restarted with the corrected, complete `partner_addrs` list, `patronictl
list` should show all nodes as Leader/Replica with matching timelines and 0 lag within
~30-60 seconds.

**Caveat**: this is a stopgap. Hand-editing `patroni.yaml` may be overwritten by the charm on a
future hook fire (e.g. any subsequent config-changed/relation-changed event that re-renders
Patroni config), so the underlying rendering bug should be fixed at the charm level.

## Environment

- Charm: `postgresql` rev 1047, channel `16/stable`
- Juju via JAAS (Prodstack7 / OpenStack), 3-unit deployment
- Patroni's DCS backend: Raft (not etcd)

## Suggested area to investigate

The charm code path that renders `raft.partner_addrs` into `patroni.yaml` on
scale-up/relation-joined for peer units. Likely a race or stale-relation-data issue where not
all peer IPs are known yet at the time the config is rendered for a given unit, and the charm
doesn't re-render it once the full peer set becomes available.

Contributor guide

Open the contributing guide

Research direction

Trace the charm code path that renders raft.partner_addrs into patroni.yaml during scale-up and peer relation events. Reproduce with a three-unit deployment, inspect each unit's patroni.yaml and Raft journal, and compare juju status with patronictl list. Done means every unit receives all N-1 peer addresses and the full Raft group forms without manual edits.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.