canonical / canonical/postgresql-operator
backend-database relation intermittently gets a narrow pg_hba.conf rule instead of 'all', due to a replica-read race in relations_user_databases_map
- Dominant language
- Python
- Stars
- 20
- Forks
- 36
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 30
Description
## Steps to reproduce
1. Deploy `postgresql` (16/stable) with 3 units, where the Patroni primary is a *different* unit than the current Juju application leader (e.g. `postgresql/67` is Patroni primary, `postgresql/66` is the Juju leader).
2. Deploy `pgbouncer` (1/stable) as a subordinate on a principal application (e.g. related via `pgbouncer:database` to some app).
3. Relate `pgbouncer:backend-database` to `postgresql:database`. This relation requests `extra-user-roles: SUPERUSER`.
4. Observe `pgbouncer`'s unit status.
## Expected behavior
`pgbouncer` initializes its `backend-database` relation successfully: it creates its auth function against the `postgres` database, using the newly created `SUPERUSER` relation role.
## Actual behavior
`pgbouncer` intermittently fails to initialize with:
```
ERROR unit.pgbouncer/.juju-log backend-database:: deferring database-created hook - Unable to initialise auth function: connection to server at "", port 5432 failed: FATAL: no pg_hba.conf entry for host "", user "relation-", database "postgres", SSL encryption
connection to server at "", port 5432 failed: FATAL: no pg_hba.conf entry for host "", user "relation-", database "postgres", no encryption
```
Inspecting `pg_hba.conf` on the Patroni primary shows the newly created relation user was granted a narrow, single-database rule instead of `all`:
```
hostssl pgbouncer relation- 0.0.0.0/0 scram-sha-256
```
instead of the expected:
```
hostssl all relation- 0.0.0.0/0 scram-sha-256
```
Since PgBouncer's auth-function bootstrap needs to connect to the `postgres` database specifically (not just the requested `pgbouncer` database), the narrow rule causes a hard connection rejection that is never retried automatically, and the unit stays `blocked`/`waiting` indefinitely (observed for 10+ minutes with no self-recovery) until some unrelated event happens to trigger a `pg_hba.conf` recompute.
## Root cause (as far as I can tell)
`relations_user_databases_map` (`src/charm.py`) computes each relation user's accessible databases via [`list_accessible_databases_for_user`](https://github.com/canonical/postgresql-operator/blob/main/lib/charms/postgresql_k8s/v0/postgresql.py), which is called with `current_host=self.is_connectivity_enabled` (effectively always `True` by default). This makes the query connect to `self.current_host` (whichever unit is running the hook, i.e. the Juju leader) instead of `self.primary_host`:
```python
host = self.current_host if current_host else None
```
Meanwhile, the relation's `create_user()` call (in `postgresql_provider.py`'s `_on_database_requested`, which is Juju-leader-gated) always writes via `self.primary_host`:
```python
host = database_host if database_host is not None else self.primary_host
```
So when the Juju application leader unit is a Patroni *replica* (not the primary), a completely normal, common topology, the `CREATE ROLE ... SUPERUSER` statement is committed on the primary, but the very next `list_accessible_databases_for_user` check (used to decide the `pg_hba.conf` scope for that same relation, moments later in the same hook execution) reads from the *replica*'s local view via streaming replication. If that replica hasn't yet applied the just-committed WAL for the new/altered role (even a very small replication-lag window), `pg_catalog.pg_user.usesuper` for that user reads back `false` there, so the narrow, database-specific `pg_hba.conf` rule gets written instead of `all`.
Critically, nothing re-triggers this computation afterward: `_on_update_status` never calls `update_config()` (only during backup-restore flows), so once a relation gets the narrow rule from this race, it stays wrong indefinitely until some *unrelated* event (leader-elected, `config-changed`, a TLS cert push, unit removal/reconfigure) happens to fire `update_config()` again. By that point replication has long since caught up, so it "self-heals" only by accident.
## Confirmed with a live reproduction
We reliably reproduced this on a live 3-unit `postgresql` cluster in a `pgbouncer` `backend-database` relation, and confirmed the workaround: forcing `update_config()` to run on all units via a harmless config change (e.g. `juju config postgresql connection_authentication_timeout=`, reverted immediately after) correctly recomputed `pg_hba.conf` to the broad `all` rule once replication had caught up, and the previously `blocked` `pgbouncer` unit recovered once its own charm reprocessed the deferred hook (in our case, triggered via `pgbouncer`'s own harmless config toggle).
## Suggested fix
`list_accessible_databases_for_user` (and `list_users`) should read from the *primary* (not `current_host`) when this check is meant to reflect the state immediately after a write that itself went to the primary. More generally, `relations_user_databases_map`'s per-relation lookup should retry/re-poll briefly if the just-created user doesn't yet show the expected privileges, rather than committing a stale, narrow `pg_hba.conf` rule on the first read.
## Versions
Operating system: ubuntu 24.04
Juju CLI: 3.6.12
Juju agent: 3.6.12
Charm revision: 16/stable, 1047
## Additional context
This was found while integrating `pgbouncer` in front of application charms other than `landscape-server` on the same PostgreSQL cluster (deploying a second, dedicated `pgbouncer` application for a different principal). It is intermittent: some deployments succeed on the very first relation attempt (if the Juju leader happens to also be the Patroni primary, or replication catches up before the read), while others fail every time until a fix like the one described above is manually applied.
Contributor guide
Research direction
Start with relations_user_databases_map in src/charm.py, then inspect list_accessible_databases_for_user and list_users in lib/charms/postgresql_k8s/v0/postgresql.py alongside _on_database_requested in postgresql_provider.py. Reproduce the leader-on-replica topology and verify the relation lookup uses primary-consistent privilege data or retries, resulting in an all pg_hba.conf rule and successful PgBouncer initialization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100