canonical / canonical/postgresql-operator
[PostgreSQL][VM][16] patroni doesn't start on unit reboot IF charm was in blocked state
- Dominant language
- Python
- Stars
- 20
- Forks
- 36
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 30
Description
## Steps to reproduce
1) deploy 1 unit + s3 integrator with foreign backup
2) restore foreign backup, PG will be in blocked state: `the S3 repository has backups from another cluster`
3) reboot PG unit: sudo reboot
## Expected behavior
Unit restarted and works well.
## Actual behavior
The event `start` is early exited to prevent DB damage:
```
unit-postgresql-0: 15:45:13 DEBUG unit.postgresql/0.juju-log tracing:Starting root trace with id='19b960d59ed6ff6fc507c6eb14375002'.
unit-postgresql-0: 15:45:13 DEBUG unit.postgresql/0.juju-log root:Emitting Juju event start.
unit-postgresql-0: 15:45:13 DEBUG unit.postgresql/0.juju-log __main__:Early exit on_start: Unit blocked
unit-postgresql-0: 15:45:13 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
```
## Versions
Operating system: 24.04
Juju CLI: 3.6.7
Juju agent: 3.6.7
Charm revision: 16/beta r789
LXD: 5.21.3-c5ae129 33110 5.21/stable/
## Log output
See above, also following update-status is early exited too:
```
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log root:Emitting Juju event update_status.
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log __main__:on_update_status early exit: Unit is in Blocked status
```
If user tried to change path in this case, the event s3 credentials_changed is being defered:
```
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log tracing:Starting root trace with id='c02422d3f62d3a3fa1c123e8c32b15ff'.
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log ops.framework:Re-emitting deferred event via PostgresqlOperatorCharm/S3Requirer[s3-parameters]/on/credentials_changed[161]>.
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log backups:Cannot initialise stanza yet.
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log ops.framework:Deferring via PostgresqlOperatorCharm/S3Requirer[s3-parameters]/on/credentials_changed[161]>.
```
## Additional context
The issue doesn't exist if charm is NOT blocked on start event: charm recovers succesfully.
P.S. we need automated test for this case, IMHO. As it looks like in PG14 update-status starts/recovers Patroni but it is changes for PG16 due to Refresh v3.
Workaround:
1) `sudo snap start charmed-postgresql.patroni`
2) `jhack fire postgresql update-status` # or wait for Juju event
Contributor guide
Assessment
This issue has not been assessed yet.