canonical / canonical/postgresql-operator
pg_wal filling the filesystem
- Dominant language
- Python
- Stars
- 20
- Forks
- 36
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 30
Description
## Steps to reproduce
1. Stop a non-primary unit
2. Wait
## Expected behavior
The "sync standby" unit WALs get pruned way before the filesystem gets full
## Actual behavior
The "sync standby" filesystem gets full - in my case the pg_wal directory was taking 1 TiB out of a 1.5 TiB filesystem.
Cluster status :
```
+ Cluster: postgresql (7527323556411144740) --+-----------+-----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+--------------+---------------+--------------+-----------+-----+-----------+
| postgresql-0 | 10.X.Y.221 | Leader | running | 902 | |
| postgresql-2 | 10.X.Y.218 | Sync Standby | streaming | 902 | 0 |
| postgresql-5 | 10.X.Y.215 | Replica | running | 901 | 2152239 |
```
## Versions
Operating system: Ubuntu 22.04.5 LTS
Juju CLI: 3.6.12-ubuntu-amd64
Juju agent: 3.6.14
Charm revision: 14/stable 936
## Additional context
My understanding is that once a secondary malfunctions (stops or gets killed or has network issues), the WALs will accumulate forever without limit. Is there a way to tell postgresql to consider the malfunctioning unit "lost" after a configurable amount of lag (in seconds or in bytes or in percent of the filesystem) to avoid filesystems getting full ?
Thanks
Contributor guide
Research direction
Start with the reported reproduction: stop a non-primary unit and observe PostgreSQL WAL growth on the sync standby, using the cluster status and version details in the issue. Trace how the operator handles stopped or lagging standbys and determine the intended recovery or pruning behavior. Done means a malfunctioning standby cannot fill its filesystem indefinitely, with coverage for the reported scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100