canonical / canonical/postgresql-operator

The charm allows a 2-node cluster but it's not functional after a failover

Open
#570 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

## Steps to reproduce

1. Prepare a MAAS provider
2. deploy the charm with 2 units by following https://charmhub.io/postgresql/docs/h-scale
`juju deploy postgresql --base ubuntu@22.04 --channel 14/stable -n 2`
3. take down the primary unit

## Expected behavior

It's either:
- keep functional after taking down one of the two units
- or prevent a two-node cluster from being deployed by making juju status blocked by suggesting 3 units instead

## Actual behavior

Similar topic with https://github.com/canonical/postgresql-operator/issues/566.

Juju status looks okay at a glance. However, the living unit doesn't say which unit is the primary at the moment.

```
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
postgres mysunbeam-controller mysunbeam/default 3.5.3 unsupported 12:17:40Z

App Version Status Scale Charm Channel Rev Exposed Message
postgresql 14.11 active 1/2 postgresql 14/stable 429 no

Unit Workload Agent Machine Public address Ports Message
postgresql/0 unknown lost 0 192.168.151.115 5432/tcp agent lost, see 'juju show-status-log postgresql/0'
postgresql/1* active idle 1 192.168.151.116 5432/tcp

Machine State Address Inst id Base AZ Message
0 down 192.168.151.115 machine-7 ubuntu@22.04 default Deployed
1 started 192.168.151.116 machine-8 ubuntu@22.04 default Deployed
```

Also, the action states the dead unit is the primary, which shouldn't be.

```
$ juju run postgresql/leader get-primary
Running operation 3 with 1 task
- task 4 on unit-postgresql-1

Waiting for task 4...
primary: postgresql/0
```

The patroni's member list cannot be fetched since the quorum of the raft was lost.
```
$ juju ssh postgresql/1 -- sudo -u snap_daemon env PATRONI_LOG_LEVEL=DEBUG patronictl -c /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml list
2024-08-05 12:20:16,176 - DEBUG - Loading configuration from file /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml
2024-08-05 12:20:21,243 - INFO - waiting on raft
2024-08-05 12:20:26,243 - INFO - waiting on raft
2024-08-05 12:20:31,244 - INFO - waiting on raft
2024-08-05 12:20:36,244 - INFO - waiting on raft
2024-08-05 12:20:41,245 - INFO - waiting on raft
2024-08-05 12:20:46,245 - INFO - waiting on raft
2024-08-05 12:20:51,246 - INFO - waiting on raft
2024-08-05 12:20:56,247 - INFO - waiting on raft
^C
Aborted!
Connection to 192.168.151.116 closed.
```

On a side note, the raft support is deprecated in patroni upstream as of 3.0.0.
https://patroni.readthedocs.io/en/latest/releases.html#version-3-0-0

## Versions

Operating system: jammy

Juju CLI: 3.5.3

Juju agent: 3.5.3

Charm revision: 14/stable 429

LXD: N/A

## Log output

Juju debug log:
[model_debug.log](https://github.com/user-attachments/files/16495641/model_debug.log)

## Additional context

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two-unit deployment from the issue and observe failover with `juju status`, `juju run postgresql/leader get-primary`, and Patroni's member-list command. No implementation files or tests are named. Done means either a two-node cluster remains functional after one unit fails, or Juju status blocks two-unit deployments and recommends three units.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
databases, devops, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.