canonical / canonical/postgresql-operator
Temporary error message: hook failed: "start" on VM recovery
- Dominant language
- Python
- Stars
- 20
- Forks
- 36
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 30
Description
Hi,
It is a followup for https://github.com/canonical/postgresql-operator/issues/566
It is more cosmetic issue, but it worth to report it and polish this case as it also may abort CI nightly tests.
## Steps to reproduce
* Deploy landscape-scalable bundle using the latest 14/stable charm revision in cluster mode:
```
> cat overlay.yaml
applications:
postgresql:
charm: ch:postgresql
channel: 14/stable
revision: 468
num_units: 3
> juju deploy landscape-scalable --overlay overlay.yaml
> # wait for all successfully deployed
```
* Check primary IP landscape in use:
```
juju exec --unit landscape-server/0 -- grep 5432 /etc/landscape/service.conf
```
* Stop Primary using `lxc stop --force juju-3088be-3` (where `juju-3088be-3` is a Primary LXC container)
* Wait for new Primary promoted and joined cluster, check new IP in use: `juju exec --unit landscape-server/0 -- grep 5432 /etc/landscape/service.conf`
* Start the manually stopped VM: `lxc start juju-3088be-3`
## Expected behavior
The newly started/restored VM joined the cluster without errors.
## Actual behavior
1) Temporary error on Juju status (it will gone after some time):

2) Mutiple Errors in Debug-log (check with LXD team the necessary systemd tuning for reliable mount point):
```
unit-postgresql-2: 13:19:53 ERROR unit.postgresql/2.juju-log Data directory not attached. Reboot unit.
unit-postgresql-2: 13:20:00 ERROR unit.postgresql/2.juju-log Data directory not attached. Reboot unit.
unit-postgresql-2: 13:20:09 ERROR unit.postgresql/2.juju-log Data directory not attached. Reboot unit.
unit-postgresql-2: 13:20:09 ERROR unit.postgresql/2.juju-log Data directory not attached. Reboot unit.
```
## Versions
Operating system: 22.04
Juju CLI: 3.5.3
Juju agent: 3.5.3
Charm revision: 468/amd
LXD: 5.0.3-80aeff7
## Log output
Juju debug log:
```
machine-3: 13:32:58 INFO juju.worker.authenticationworker "machine-3" key updater worker started
unit-postgresql-1: 13:32:58 INFO juju.worker.uniter unit "postgresql/1" started
unit-postgresql-1: 13:32:58 INFO juju.worker.uniter hooks are retried true
unit-postgresql-1: 13:32:58 INFO juju.worker.uniter reboot detected; triggering implicit start hook to notify charm
unit-postgresql-1: 13:32:59 DEBUG unit.postgresql/1.juju-log ops 2.16.0 up and running.
unit-postgresql-1: 13:32:59 INFO unit.postgresql/1.juju-log Running legacy hooks/start.
unit-postgresql-1: 13:32:59 DEBUG unit.postgresql/1.juju-log ops 2.16.0 up and running.
unit-postgresql-1: 13:32:59 DEBUG unit.postgresql/1.juju-log Charm called itself via hooks/start.
unit-postgresql-1: 13:32:59 DEBUG unit.postgresql/1.juju-log Legacy hooks/start exited with status 0.
unit-postgresql-1: 13:33:00 INFO unit.postgresql/1.juju-log Starting cluster topology observer process
unit-postgresql-1: 13:33:00 INFO unit.postgresql/1.juju-log Started cluster topology observer process with PID 786
unit-postgresql-1: 13:33:00 DEBUG unit.postgresql/1.juju-log no relation on 'tracing': tracing not ready
unit-postgresql-1: 13:33:00 DEBUG unit.postgresql/1.juju-log Emitting Juju event start.
unit-postgresql-1: 13:33:00 DEBUG unit.postgresql/1.juju-log Deferring .
unit-postgresql-1: 13:33:00 ERROR unit.postgresql/1.juju-log Data directory not attached. Reboot unit.
```
## Additional context
All those errors are cosmetic, the PostgreSQL cluster is resurected, but it takes some time (could wait the next update-status).
We can try to polish UX here.
Also, on jhack tail output I see many `start` events EVEN if charm is not restarting container:

No ideas, why...
Contributor guide
Assessment
This issue has not been assessed yet.