canonical / canonical/postgresql-k8s-operator

Cannot deploy more than 10 units (max_wal_senders limit)

Open
#895 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
15
Forks
33
Avg merge
1d 3h
Merged PRs (30d)
41

Description

[Same issue in the VM](https://github.com/canonical/postgresql-operator/issues/801)

## Steps to reproduce

1. `juju deploy postgresql-k8s --channel edge --config profile=testing -n10` and wait for them to stabilize
2. `juju scale-application postgresql-k8s 11`

## Expected behavior
Unit is added successfully therefore cluster consists of 11 units.

## Actual behavior

Several unit stuck in `awaiting for member to start`.
Image

## Versions

Operating system: Ubuntu 24.04.2 LTS

Juju CLI: 3.6.4-genericlinux-amd64

Juju agent: 3.6.4

Charm revision: 527

microk8s: MicroK8s v1.32.2 revision 7750

## Log output

Juju debug log: [tmp.log](https://github.com/user-attachments/files/19363548/tmp.log)

```
root@postgresql-k8s-1:/# pebble logs postgresql
2025-03-20T10:15:54.468Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13134" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:15:54.468Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:15:54.469Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:15:58.074Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13146" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:15:58.074Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:15:58.075Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:04.719Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13157" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:16:04.719Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:16:04.720Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:07.524Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:07.524Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:14.056Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:14.057Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:17.624Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:17.625Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:24.717Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13205" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:16:24.717Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:16:24.717Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:27.947Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:27.947Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:34.732Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13227" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:16:34.732Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:16:34.733Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:37.644Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:37.645Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:44.359Z [postgresql] pg_basebackup: error: connection to server at "postgresql-k8s-2.postgresql-k8s-endpoints" (10.1.142.183), port 5432 failed: FATAL: number of requested standby connections exceeds max_wal_senders (currently 10)
2025-03-20T10:16:44.360Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
2025-03-20T10:16:48.008Z [postgresql] pg_basebackup: error: could not send replication command "CREATE_REPLICATION_SLOT "pg_basebackup_13256" TEMPORARY PHYSICAL RESERVE_WAL": ERROR: all replication slots are in use
2025-03-20T10:16:48.008Z [postgresql] HINT: Free one or increase max_replication_slots.
2025-03-20T10:16:48.010Z [postgresql] pg_basebackup: removing data directory "/var/lib/postgresql/data/pgdata"
```

## Additional context

I think including `max_wal_senders` and `max_replication_slots` config options will resolve this issue and add an abiltiy to include charm checks based on that value.

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Start by reproducing the issue with the listed Juju deployment and scale commands, then inspect the PostgreSQL logs for max_wal_senders and max_replication_slots failures. Done means scaling from 10 to 11 units completes without units remaining in “awaiting for member to start”.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, postgresql
Domain
databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.