canonical / canonical/postgresql-operator

pg_wal filling the filesystem

Open
#1,448 9 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

## Steps to reproduce

1. Stop a non-primary unit
2. Wait

## Expected behavior
The "sync standby" unit WALs get pruned way before the filesystem gets full

## Actual behavior

The "sync standby" filesystem gets full - in my case the pg_wal directory was taking 1 TiB out of a 1.5 TiB filesystem.
Cluster status :
```
+ Cluster: postgresql (7527323556411144740) --+-----------+-----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+--------------+---------------+--------------+-----------+-----+-----------+
| postgresql-0 | 10.X.Y.221 | Leader | running | 902 | |
| postgresql-2 | 10.X.Y.218 | Sync Standby | streaming | 902 | 0 |
| postgresql-5 | 10.X.Y.215 | Replica | running | 901 | 2152239 |
```

## Versions

Operating system: Ubuntu 22.04.5 LTS

Juju CLI: 3.6.12-ubuntu-amd64

Juju agent: 3.6.14

Charm revision: 14/stable 936

## Additional context

My understanding is that once a secondary malfunctions (stops or gets killed or has network issues), the WALs will accumulate forever without limit. Is there a way to tell postgresql to consider the malfunctioning unit "lost" after a configurable amount of lag (in seconds or in bytes or in percent of the filesystem) to avoid filesystems getting full ?

Thanks

Contributor guide

Open the contributing guide

Research direction

Start with the reported reproduction: stop a non-primary unit and observe PostgreSQL WAL growth on the sync standby, using the cluster status and version details in the issue. Trace how the operator handles stopped or lagging standbys and determine the intended recovery or pruning behavior. Done means a malfunctioning standby cannot fill its filesystem indefinitely, with coverage for the reported scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.