canonical / canonical/postgresql-operator

PostgreSQL cluster split-brain when network in unstable

Open
#712 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

## Steps to reproduce

Prerequisites:
1) install Ubuntu from `ubuntu/jammy64` box on Vagrant(2.4.3) + VirtualBox(7.0.22-165102~Ubuntu~jammy).
2) install juju inside ubuntu VM
3) install lxd inside Ubuntu VM
4) install PostgreSQL charm according to [tutorial](https://charmhub.io/postgresql/docs/t-overview), scale it up to 2 replicas

Initial status of the system:
```
Model Controller Cloud/Region Version SLA Timestamp
postgresql localhost-localhost localhost/localhost 3.4.6 unsupported 07:53:40Z

App Version Status Scale Charm Channel Rev Exposed Message
postgresql 14.12 active 3 postgresql 14/stable 468 no

Unit Workload Agent Machine Public address Ports Message
postgresql/0 active idle 0 10.232.17.70 5432/tcp
postgresql/1* active idle 1 10.232.17.207 5432/tcp
postgresql/2 active idle 2 10.232.17.155 5432/tcp Primary

Machine State Address Inst id Base AZ Message
0 started 10.232.17.70 juju-e99525-0 ubuntu@22.04 Running
1 started 10.232.17.207 juju-e99525-1 ubuntu@22.04 Running
2 started 10.232.17.155 juju-e99525-2 ubuntu@22.04 Running
```

1. Connect [serial-vault](https://github.com/canonical/serial-vault) app to DB
2. Run [serial-vault-perf-tests](https://github.com/alex-ramanau/serial-vault-perf-tests) on the rate **200 RPS**
3. Wait 10-15 minute to be sure that app+DB work as expected (200 OKs are returned, and there are enough resources - CPU/RAM for this workload)
4. Break partially network on PostgreSQL primary:
```
juju ssh postgresql/2
sudo tc qdisc add dev eth0 root netem loss 80%
```
5. Wait 30-60 minutes and inspect cluster behavior

## Expected behavior

The failed primary node is detected and kicked out of the cluster. The cluster works well in degraded mode with 1 replica until manual intervention.

## Actual behavior

Cluster had 2 primary nodes, see juju status:
```
Model Controller Cloud/Region Version SLA Timestamp
postgresql localhost-localhost localhost/localhost 3.4.6 unsupported 21:43:32Z

App Version Status Scale Charm Channel Rev Exposed Message
postgresql 14.12 active 3 postgresql 14/stable 468 no

Unit Workload Agent Machine Public address Ports Message
postgresql/0 active idle 0 10.232.17.70 5432/tcp Primary
postgresql/1* active idle 1 10.232.17.207 5432/tcp
postgresql/2 active idle 2 10.232.17.155 5432/tcp Primary

Machine State Address Inst id Base AZ Message
0 started 10.232.17.70 juju-e99525-0 ubuntu@22.04 Running
1 started 10.232.17.207 juju-e99525-1 ubuntu@22.04 Running
2 started 10.232.17.155 juju-e99525-2 ubuntu@22.04 Running
```

PostgreSQL cluster nodes were also reported two DB masters for postgresql/2 and postgresql/0:
```
$ psql -h 10.232.17.70 -U operator -p 5432
postgres=# select pg_is_in_recovery();
pg_is_in_recovery
-------------------
f

# with long delay due to slow network
$ psql -h 10.232.17.155 -U operator -p 5432
postgres=# select pg_is_in_recovery();
pg_is_in_recovery
-------------------
f
```

So it looks like a cluster split-brain.

## Versions

Operating system: Ubuntu 22.04.4 LTS

Juju CLI: 3.6.1-genericlinux-amd64

Juju agent: 3.4.6

Charm revision: 468

LXD: 5.0.4

## Log output
I've attached the juju, PostgreSQL and patroni logs:
[database_logs_new_primary.tar.gz](https://github.com/user-attachments/files/18329597/database_logs_new_primary.tar.gz)
[database_logs_old_primary.tar.gz](https://github.com/user-attachments/files/18329598/database_logs_old_primary.tar.gz)
[juju_status_two_primaries.log](https://github.com/user-attachments/files/18329599/juju_status_two_primaries.log)
[juju-debug-log.log](https://github.com/user-attachments/files/18329600/juju-debug-log.log)

## Additional context
Please note, that cluster self-repaired after rebooting the problem node.

The previous run on my local machine has a different outcome:
> One ofPostgreSQL replicas was promoted to primary. Primary became the replica and status is green from charm point of view. However, actual replica status is broken

I'll reduce the case using pgbench and test it again soon.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.