canonical / canonical/postgresql-operator

Temporary error message: hook failed: "start" on VM recovery

Open
#618 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

Hi,

It is a followup for https://github.com/canonical/postgresql-operator/issues/566
It is more cosmetic issue, but it worth to report it and polish this case as it also may abort CI nightly tests.

## Steps to reproduce
* Deploy landscape-scalable bundle using the latest 14/stable charm revision in cluster mode:
```
> cat overlay.yaml
applications:
postgresql:
charm: ch:postgresql
channel: 14/stable
revision: 468
num_units: 3

> juju deploy landscape-scalable --overlay overlay.yaml
> # wait for all successfully deployed
```
* Check primary IP landscape in use:
```
juju exec --unit landscape-server/0 -- grep 5432 /etc/landscape/service.conf
```
* Stop Primary using `lxc stop --force juju-3088be-3` (where `juju-3088be-3` is a Primary LXC container)
* Wait for new Primary promoted and joined cluster, check new IP in use: `juju exec --unit landscape-server/0 -- grep 5432 /etc/landscape/service.conf`
* Start the manually stopped VM: `lxc start juju-3088be-3`

## Expected behavior
The newly started/restored VM joined the cluster without errors.

## Actual behavior
1) Temporary error on Juju status (it will gone after some time):
![Screenshot from 2024-09-12 13-35-41](https://github.com/user-attachments/assets/9119d45c-14cc-470f-abc2-b68e1eaa57d5)

2) Mutiple Errors in Debug-log (check with LXD team the necessary systemd tuning for reliable mount point):
```
unit-postgresql-2: 13:19:53 ERROR unit.postgresql/2.juju-log Data directory not attached. Reboot unit.
unit-postgresql-2: 13:20:00 ERROR unit.postgresql/2.juju-log Data directory not attached. Reboot unit.
unit-postgresql-2: 13:20:09 ERROR unit.postgresql/2.juju-log Data directory not attached. Reboot unit.
unit-postgresql-2: 13:20:09 ERROR unit.postgresql/2.juju-log Data directory not attached. Reboot unit.
```

## Versions

Operating system: 22.04
Juju CLI: 3.5.3
Juju agent: 3.5.3
Charm revision: 468/amd
LXD: 5.0.3-80aeff7

## Log output
Juju debug log:
```
machine-3: 13:32:58 INFO juju.worker.authenticationworker "machine-3" key updater worker started
unit-postgresql-1: 13:32:58 INFO juju.worker.uniter unit "postgresql/1" started
unit-postgresql-1: 13:32:58 INFO juju.worker.uniter hooks are retried true
unit-postgresql-1: 13:32:58 INFO juju.worker.uniter reboot detected; triggering implicit start hook to notify charm
unit-postgresql-1: 13:32:59 DEBUG unit.postgresql/1.juju-log ops 2.16.0 up and running.
unit-postgresql-1: 13:32:59 INFO unit.postgresql/1.juju-log Running legacy hooks/start.
unit-postgresql-1: 13:32:59 DEBUG unit.postgresql/1.juju-log ops 2.16.0 up and running.
unit-postgresql-1: 13:32:59 DEBUG unit.postgresql/1.juju-log Charm called itself via hooks/start.
unit-postgresql-1: 13:32:59 DEBUG unit.postgresql/1.juju-log Legacy hooks/start exited with status 0.
unit-postgresql-1: 13:33:00 INFO unit.postgresql/1.juju-log Starting cluster topology observer process
unit-postgresql-1: 13:33:00 INFO unit.postgresql/1.juju-log Started cluster topology observer process with PID 786
unit-postgresql-1: 13:33:00 DEBUG unit.postgresql/1.juju-log no relation on 'tracing': tracing not ready
unit-postgresql-1: 13:33:00 DEBUG unit.postgresql/1.juju-log Emitting Juju event start.
unit-postgresql-1: 13:33:00 DEBUG unit.postgresql/1.juju-log Deferring .
unit-postgresql-1: 13:33:00 ERROR unit.postgresql/1.juju-log Data directory not attached. Reboot unit.
```

## Additional context
All those errors are cosmetic, the PostgreSQL cluster is resurected, but it takes some time (could wait the next update-status).
We can try to polish UX here.

Also, on jhack tail output I see many `start` events EVEN if charm is not restarting container:
![image](https://github.com/user-attachments/assets/e8d4e335-7125-4b1a-90bf-22c415e4d435)
No ideas, why...

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.