cloudnative-pg / cloudnative-pg/cloudnative-pg

[Bug]: Orphaned logical replication slots on demoted primary cause WAL accumulation after switchover

Open
#9,969 7 comments 11 reactions 1 assignee Claimed by @gbartolini View on GitHub
Stale triage
Dominant language
Go
Stars
9.3k
Forks
759
Avg merge
2d 6h
Merged PRs (30d)
44

Description

### Is there an existing issue already for this bug?

- [x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.

### I have read the troubleshooting guide

- [x] I have read the troubleshooting guide and I think this is a new bug.

### I am running a supported version of CloudNativePG

- [x] I have read the troubleshooting guide and I think this is a new bug.

### Contact Details

_No response_

### Version

1.27 (latest patch)

### What version of Kubernetes are you using?

1.35

### What is your Kubernetes environment?

Cloud: Google GKE

### How did you install the operator?

Helm

### What happened?

When using synchronizeLogicalDecoding: true with PostgreSQL 17's native sync_replication_slots, a switchover leaves orphaned logical replication slots on the demoted primary (now replica) that
cause unbounded WAL accumulation.

The slot sync worker on the new replica continuously fails with:
exiting from slot synchronization because same name slot "firenote_reporting_sub" already exists on the standby

This occurs because the demoted primary's original slot has synced=false (it was created locally when that node was the primary), and PostgreSQL's slot sync worker refuses to overwrite slots
that aren't marked as synced=true.

How to reproduce

1. Create a CNPG cluster with instances: 2 and synchronizeLogicalDecoding: true
2. Create a logical replication subscription that connects to the primary (creates a slot with synced=false on primary, synced=true on replica)
3. Perform a switchover (kubectl cnpg promote )
4. Observe:
- The new primary has the slot with synced=true (was the synced copy)
- The new replica has the slot with synced=false (was the original)
- The slot sync worker fails repeatedly on the new replica
- WAL files accumulate on the new replica because the stale slot's restart_lsn is never advanced

### Expected behavior

After a switchover, logical replication slots should be properly synchronized, and WAL files should not accumulate on replicas due to stale slot positions.

### Actual behavior

The new replica retains its original slot with synced=false. The slot sync worker cannot overwrite it, so:
- The slot's restart_lsn is never updated
- WAL files accumulate (we observed 17GB+ on replicas vs ~600MB on primary)
- The slot sync worker logs errors every 10 seconds indefinitely

Slot state comparison after switchover:
```
Node Role synced restart_lsn
───────────────────────────────────────────────────────────
node-1 Primary true Current (8/88045520)
node-2 Replica false Stale (4/3701C680) — 17GB behind
```

### Root cause

PostgreSQL 17's slot synchronization only manages slots where synced=true. The synced column is:
- Set to true only when the slot sync worker creates a slot
- Set to false when created locally (by a subscriber or pg_create_logical_replication_slot())
- Read-only — cannot be modified

When a primary with a local logical slot (synced=false) is demoted to replica:
1. The slot remains with synced=false
2. The slot sync worker refuses to update it
3. The orphaned slot holds back WAL retention indefinitely

This is acknowledged as a PostgreSQL limitation in the https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg161150.html, where the suggested workaround is for users to manually
drop the slot on the new standby.

### Proposed solution

CNPG should handle this during switchover/failover by dropping logical replication slots with synced=false on the demoted primary after it rejoins as a replica. This would allow the
slot sync worker to recreate the slot with synced=true and proper LSN positions.

### Workaround

Manually drop the orphaned slot on the replica if it has synced=false (Could be set up as a k8s cron job):
```
SELECT pg_drop_replication_slot(slot_name)
FROM pg_replication_slots
WHERE slot_name = ''
AND synced = false
AND pg_is_in_recovery();
```

### Cluster resource

```shell
Relevant cluster spec

primaryUpdateMethod: switchover
replicationSlots:
highAvailability:
synchronizeLogicalDecoding: true
postgresql:
parameters:
wal_level: 'logical'
hot_standby_feedback: 'on'
sync_replication_slots: 'on'
```

### Relevant log output

```shell
On the original primary pod that has been demoted (switchover), we see these logs every 10s

{
backend_type: "slotsync worker"
error_severity: "ERROR"
message: "exiting from slot synchronization because same name slot "" already exists on the standby"
}
```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.