canonical / canonical/postgresql-operator

Charm does not expose remove_data_directory_on_diverged_timelines - diverged replicas loop forever

Open
#1,728 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

## Summary

`patroni.dynamic.json` hardcodes `remove_data_directory_on_diverged_timelines: false`. When a replica diverges timelines (e.g. after repeated failovers), patroni refuses to wipe and re-clone the data directory. Instead it loops forever trying to replay WAL that will never converge, accumulating WAL in `data/logs` indefinitely. There is no way to configure this via a Juju config option.

## Observed behaviour

After 77 failovers/restores in STG, `postgresql/58` was stuck at TL1 while the cluster was at TL77. The unit had accumulated 4.8 GB of WAL in `data/logs` from failed `restore_command` attempts over 24+ days. Patronictl showed the unit in `starting` state with `unknown` lag - it never self-healed.

The charm's unit status showed `blocked: failed to initialize stanza, check your S3 settings` - a secondary symptom from the charm attempting stanza-init while postgres was not yet accepting connections. S3 credentials and the bucket were healthy throughout.

## Expected behaviour

The charm should expose `remove_data_directory_on_diverged_timelines` as a Juju config option. Setting it to `true` allows patroni to automatically wipe and re-clone a diverged replica from the primary, self-healing without operator intervention.

## Steps to reproduce

1. Deploy a multi-unit `charmed-postgresql` cluster with pgbackrest S3 backup
2. Trigger enough failovers/restores to advance the cluster timeline significantly
3. Take a replica offline long enough for its WAL to diverge from the primary timeline
4. Observe the replica loops in `starting` state with `unknown` lag indefinitely - patroni will not wipe and re-clone

## Impact

- Replicas that diverge timelines require manual `patronictl reinit --force` intervention to recover
- The `blocked: failed to initialize stanza` charm status is a misleading secondary symptom - the real problem is the replica never coming up, not S3
- We hit this in STG after 77 timeline advances; any long-running cluster that experiences failovers is at risk

## Workaround

Manual intervention required:

```bash
snap run charmed-postgresql.patronictl \
-c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml \
reinit postgresql --force
```

## Environment

- Charm: `charmed-postgresql` (machine operator, not k8s)
- Juju controller: JAAS / Prodstack7
- PostgreSQL: Patroni-managed cluster, 3-unit

## Suggested fix

Expose `remove_data_directory_on_diverged_timelines` as a Juju config option (default `false` to preserve existing behaviour). When set to `true`, include it in `patroni.dynamic.json` so patroni will automatically wipe and re-clone diverged replicas.

Tracked: https://warthogs.atlassian.net/browse/LNDENG-4481

Contributor guide

Open the contributing guide

Research direction

Locate patroni.dynamic.json and the charm's Juju config definitions, then trace how existing Patroni settings are rendered. Verify that the new option defaults to false and that setting it to true is included in patroni.dynamic.json, allowing diverged replicas to re-clone while preserving current behavior by default.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.