cloudnative-pg / cloudnative-pg/cloudnative-pg
[Bug]: Orphaned logical replication slots on demoted primary cause WAL accumulation after switchover
- Dominant language
- Go
- Stars
- 9.3k
- Forks
- 759
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 44
Description
### Is there an existing issue already for this bug?
- [x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.
### I have read the troubleshooting guide
- [x] I have read the troubleshooting guide and I think this is a new bug.
### I am running a supported version of CloudNativePG
- [x] I have read the troubleshooting guide and I think this is a new bug.
### Contact Details
_No response_
### Version
1.27 (latest patch)
### What version of Kubernetes are you using?
1.35
### What is your Kubernetes environment?
Cloud: Google GKE
### How did you install the operator?
Helm
### What happened?
When using synchronizeLogicalDecoding: true with PostgreSQL 17's native sync_replication_slots, a switchover leaves orphaned logical replication slots on the demoted primary (now replica) that
cause unbounded WAL accumulation.
The slot sync worker on the new replica continuously fails with:
exiting from slot synchronization because same name slot "firenote_reporting_sub" already exists on the standby
This occurs because the demoted primary's original slot has synced=false (it was created locally when that node was the primary), and PostgreSQL's slot sync worker refuses to overwrite slots
that aren't marked as synced=true.
How to reproduce
1. Create a CNPG cluster with instances: 2 and synchronizeLogicalDecoding: true
2. Create a logical replication subscription that connects to the primary (creates a slot with synced=false on primary, synced=true on replica)
3. Perform a switchover (kubectl cnpg promote )
4. Observe:
- The new primary has the slot with synced=true (was the synced copy)
- The new replica has the slot with synced=false (was the original)
- The slot sync worker fails repeatedly on the new replica
- WAL files accumulate on the new replica because the stale slot's restart_lsn is never advanced
### Expected behavior
After a switchover, logical replication slots should be properly synchronized, and WAL files should not accumulate on replicas due to stale slot positions.
### Actual behavior
The new replica retains its original slot with synced=false. The slot sync worker cannot overwrite it, so:
- The slot's restart_lsn is never updated
- WAL files accumulate (we observed 17GB+ on replicas vs ~600MB on primary)
- The slot sync worker logs errors every 10 seconds indefinitely
Slot state comparison after switchover:
```
Node Role synced restart_lsn
───────────────────────────────────────────────────────────
node-1 Primary true Current (8/88045520)
node-2 Replica false Stale (4/3701C680) — 17GB behind
```
### Root cause
PostgreSQL 17's slot synchronization only manages slots where synced=true. The synced column is:
- Set to true only when the slot sync worker creates a slot
- Set to false when created locally (by a subscriber or pg_create_logical_replication_slot())
- Read-only — cannot be modified
When a primary with a local logical slot (synced=false) is demoted to replica:
1. The slot remains with synced=false
2. The slot sync worker refuses to update it
3. The orphaned slot holds back WAL retention indefinitely
This is acknowledged as a PostgreSQL limitation in the https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg161150.html, where the suggested workaround is for users to manually
drop the slot on the new standby.
### Proposed solution
CNPG should handle this during switchover/failover by dropping logical replication slots with synced=false on the demoted primary after it rejoins as a replica. This would allow the
slot sync worker to recreate the slot with synced=true and proper LSN positions.
### Workaround
Manually drop the orphaned slot on the replica if it has synced=false (Could be set up as a k8s cron job):
```
SELECT pg_drop_replication_slot(slot_name)
FROM pg_replication_slots
WHERE slot_name = ''
AND synced = false
AND pg_is_in_recovery();
```
### Cluster resource
```shell
Relevant cluster spec
primaryUpdateMethod: switchover
replicationSlots:
highAvailability:
synchronizeLogicalDecoding: true
postgresql:
parameters:
wal_level: 'logical'
hot_standby_feedback: 'on'
sync_replication_slots: 'on'
```
### Relevant log output
```shell
On the original primary pod that has been demoted (switchover), we see these logs every 10s
{
backend_type: "slotsync worker"
error_severity: "ERROR"
message: "exiting from slot synchronization because same name slot "" already exists on the standby"
}
```
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Assessment
This issue has not been assessed yet.