canonical / canonical/postgresql-k8s-operator
Cannot deploy more than 10 units (max_wal_senders limit)
- Dominant language
- Python
- Stars
- 15
- Forks
- 33
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 41
Description
[Same issue in the VM](https://github.com/canonical/postgresql-operator/issues/801)
## Steps to reproduce
1. `juju deploy postgresql-k8s --channel edge --config profile=testing -n10` and wait for them to stabilize
2. `juju scale-application postgresql-k8s 11`
## Expected behavior
Unit is added successfully therefore cluster consists of 11 units.
## Actual behavior
Several unit stuck in `awaiting for member to start`.
## Versions
Operating system: Ubuntu 24.04.2 LTS
Juju CLI: 3.6.4-genericlinux-amd64
Juju agent: 3.6.4
Charm revision: 527
microk8s: MicroK8s v1.32.2 revision 7750
## Log output
Juju debug log: [tmp.log](https://github.com/user-attachments/files/19363548/tmp.log)
```
root@postgresql-k8s-1:/# pebble logs postgresql
2025-03-20T10:15:54.468Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13134" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:15:54.468Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:15:54.469Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:15:58.074Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13146" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:15:58.074Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:15:58.075Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:04.719Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13157" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:16:04.719Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:16:04.720Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:07.524Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:07.524Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:14.056Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:14.057Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:17.624Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:17.625Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:24.717Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13205" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:16:24.717Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:16:24.717Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:27.947Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:27.947Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:34.732Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13227" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:16:34.732Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:16:34.733Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:37.644Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:37.645Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:44.359Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:44.360Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:48.008Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13256" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:16:48.008Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:16:48.010Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
```
## Additional context
I think including `max_wal_senders` and `max_replication_slots` config options will resolve this issue and add an abiltiy to include charm checks based on that value.
Contributor guide
Research direction
No source file or test is named. Start by reproducing the issue with the listed Juju deployment and scale commands, then inspect the PostgreSQL logs for max_wal_senders and max_replication_slots failures. Done means scaling from 10 to 11 units completes without units remaining in “awaiting for member to start”.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, postgresql
- Domain
- databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100