canonical / canonical/postgresql-k8s-operator

PG16-K8s cannot be deployed to Canonical K8s. Error: calling self.fix_leader_annotation() (PG14-K8s works)

Closed
#1,009 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
15
Forks
33
Avg merge
1d 3h
Merged PRs (30d)
41

Description

## Steps to reproduce
1. Launch LXD with enough disk/CPU/RAM: `lxc launch --vm ubuntu:24.04 -c limits.cpu=12 -c limits.memory=32GiB -d root,size=200GiB myc8s`
2. Deploy Canonical K8s inside newly started VM using https://canonical-charmed-postgresql-k8s.readthedocs-hosted.com/16/how-to/deploy/canonical-k8s/
3. try to deploy PG16: `juju deploy postgresql-k8s --channel 16/edge --trust` (r613 is failing). PG14 woks well (r495)!

## Expected behavior
PG16 deployed without errors

## Actual behavior
Charm stuck in `awaiting for cluster to start`:
```
root@2xenrico:~# juju status
Model Controller Cloud/Region Version SLA Timestamp
test16 ck8s ck8s 3.6.7 unsupported 16:35:03Z

App Version Status Scale Charm Channel Rev Address Exposed Message
postgresql-k8s 16.9 waiting 1 postgresql-k8s 16/edge 613 10.152.183.219 no awaiting for cluster to start

Unit Workload Agent Address Ports Message
postgresql-k8s/0* waiting executing 10.1.0.172 awaiting for cluster to start
```

## Versions

Operating system: 24.04
Juju CLI: 3.6.7
Juju agent: 3.6.7
Charm revision: postgresql-k8s 16/edge r613
Canonical K8s: k8s v1.32.5 3581 1.32-classic/stable canonical✓ classic

## Log output
```
unit-postgresql-k8s-0: 16:42:03 DEBUG unit.postgresql-k8s/0.juju-log ops 2.22.0 up and running.
unit-postgresql-k8s-0: 16:42:03 WARNING unit.postgresql-k8s/0.juju-log /var/lib/juju/agents/unit-postgresql-k8s-0/charm/src/charm.py:220: DeprecationWarning: JujuVersion.from_environ() is deprecated, use self.model.juju_version instead
unit-postgresql-k8s-0: 16:42:03 DEBUG unit.postgresql-k8s/0.juju-log no relation on 'tracing': tracing not ready
unit-postgresql-k8s-0: 16:42:03 DEBUG unit.postgresql-k8s/0.juju-log Starting root trace with id='b46b424fd89d9f9b71781b7ed3d9761a'.
unit-postgresql-k8s-0: 16:42:03 DEBUG unit.postgresql-k8s/0.juju-log Re-emitting deferred event via PostgresqlOperatorCharm/on/postgresql_pebble_ready[26]>.
unit-postgresql-k8s-0: 16:42:03 DEBUG unit.postgresql-k8s/0.juju-log Building PostgreSQL parameters for profile='production' and available_memory=33541885952
unit-postgresql-k8s-0: 16:42:03 INFO unit.postgresql-k8s/0.juju-log Updating Patroni config file
unit-postgresql-k8s-0: 16:42:04 DEBUG unit.postgresql-k8s/0.juju-log Early exit update_config: Workload not started yet
unit-postgresql-k8s-0: 16:42:04 INFO unit.postgresql-k8s/0.juju-log Updated health checks
unit-postgresql-k8s-0: 16:42:04 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:05 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:06 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:07 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:08 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:09 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:10 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:11 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:12 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:13 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-0.postgresql-k8s-endpoints:8008
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Deferring on_postgresql_pebble_ready: Waiting for cluster to start
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Deferring via PostgresqlOperatorCharm/on/postgresql_pebble_ready[26]>.
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Re-emitting deferred event .
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Defer on_config_changed: cluster not initialised yet
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Deferring .
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Re-emitting deferred event via PostgresqlOperatorCharm/on/database_peers_relation_changed[71]>.
unit-postgresql-k8s-0: 16:42:14 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:15 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:16 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:17 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:18 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:19 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:20 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:21 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:22 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:23 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log Starting new HTTP connection (1): postgresql-k8s-primary.test16.svc.cluster.local:8008
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log Deferring on_postgresql_pebble_ready: Waiting for primary endpoint to be ready
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log Deferring via PostgresqlOperatorCharm/on/database_peers_relation_changed[71]>.
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log _initialized_cluster failed on _peer_relation_changed
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log Emitting Juju event update_status.
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log pgdata free disk space: 1003884544 out of 1020702720, ratio of 0.9835229438792913
unit-postgresql-k8s-0: 16:42:24 ERROR unit.postgresql-k8s/0.juju-log calling self.fix_leader_annotation()
unit-postgresql-k8s-0: 16:42:24 DEBUG unit.postgresql-k8s/0.juju-log on_update_status early exit: Unit is in Blocked/Waiting status
unit-postgresql-k8s-0: 16:42:24 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
```

Contributor guide

Open the contributing guide

Research direction

Start with the Canonical K8s deployment steps and the charm log around self.fix_leader_annotation(), then compare the failing PostgreSQL 16 deployment with the working PostgreSQL 14 case. Reproduce the reported deployment using the listed Juju, charm, and Kubernetes versions; done means PostgreSQL 16 reaches a running cluster instead of waiting for the cluster to start.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, postgresql, python
Domain
databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.