canonical / canonical/postgresql-operator

[PostgreSQL][VM][16] patroni doesn't start on unit reboot IF charm was in blocked state

Open
#986 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

## Steps to reproduce

1) deploy 1 unit + s3 integrator with foreign backup
2) restore foreign backup, PG will be in blocked state: `the S3 repository has backups from another cluster`
3) reboot PG unit: sudo reboot

## Expected behavior

Unit restarted and works well.

## Actual behavior

The event `start` is early exited to prevent DB damage:
```
unit-postgresql-0: 15:45:13 DEBUG unit.postgresql/0.juju-log tracing:Starting root trace with id='19b960d59ed6ff6fc507c6eb14375002'.
unit-postgresql-0: 15:45:13 DEBUG unit.postgresql/0.juju-log root:Emitting Juju event start.
unit-postgresql-0: 15:45:13 DEBUG unit.postgresql/0.juju-log __main__:Early exit on_start: Unit blocked
unit-postgresql-0: 15:45:13 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
```

## Versions

Operating system: 24.04
Juju CLI: 3.6.7
Juju agent: 3.6.7

Charm revision: 16/beta r789
LXD: 5.21.3-c5ae129 33110 5.21/stable/

## Log output
See above, also following update-status is early exited too:
```
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log root:Emitting Juju event update_status.
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log __main__:on_update_status early exit: Unit is in Blocked status
```
If user tried to change path in this case, the event s3 credentials_changed is being defered:
```
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log tracing:Starting root trace with id='c02422d3f62d3a3fa1c123e8c32b15ff'.
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log ops.framework:Re-emitting deferred event via PostgresqlOperatorCharm/S3Requirer[s3-parameters]/on/credentials_changed[161]>.
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log backups:Cannot initialise stanza yet.
unit-postgresql-0: 15:50:04 DEBUG unit.postgresql/0.juju-log ops.framework:Deferring via PostgresqlOperatorCharm/S3Requirer[s3-parameters]/on/credentials_changed[161]>.
```

## Additional context

The issue doesn't exist if charm is NOT blocked on start event: charm recovers succesfully.
P.S. we need automated test for this case, IMHO. As it looks like in PG14 update-status starts/recovers Patroni but it is changes for PG16 due to Refresh v3.

Workaround:
1) `sudo snap start charmed-postgresql.patroni`
2) `jhack fire postgresql update-status` # or wait for Juju event

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.