cloudnative-pg / cloudnative-pg/cloudnative-pg

[Bug]: Replica can remain healthy after unrecoverable WAL replay loop

Open
#11,134 1 comment 1 reaction 1 assignee Claimed by @gbartolini View on GitHub
bug :bug: triage
Dominant language
Go
Stars
9.3k
Forks
759
Avg merge
2d 6h
Merged PRs (30d)
44

Description

### Is there an existing issue already for this bug?

- [x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.

### I have read the troubleshooting guide

- [x] I have read the troubleshooting guide and I think this is a new bug.

### I am running a supported version of CloudNativePG

- [x] I have read the troubleshooting guide and I think this is a new bug.

### Contact Details

schneider@ardentperf.com

### Version

1.30.0

### What version of Kubernetes are you using?

1.36

### What is your Kubernetes environment?

Cloud: Azure AKS

### How did you install the operator?

YAML manifest

### What happened?

A replica can enter an unrecoverable WAL replay failure loop while CNPG continues to treat the instance as healthy enough for normal operation. In this state the instance repeatedly logs WAL replay errors at the same LSN, but the operator does not get a hard status signal from the instance manager.

Reproducer: https://github.com/ardentperf/bug-repro-wal-restore

Observed impact:

- the broken replica becomes Ready even though WAL replay is stuck
- the pod remains eligible for read traffic through normal service endpoints
- the replica remains a possible failover/promotion target because `/pg/status` does not report an error
- diagnosis currently depends on inspecting instance logs directly

Expected behavior:

When the instance manager detects repeated unrecoverable WAL replay errors with no replay progress, it should mark the instance unhealthy through existing readiness and status paths. CNPG should stop routing traffic to the replica and existing target-selection logic should avoid it.

Known log patterns include:

- `record with incorrect prev-link ... at ...`
- `contrecord is requested by ...`

A future enhancement may be useful to surface "user action needed" conditions directly on the Cluster object, but operator and instance logs are sufficient for the initial fix.

### Cluster resource

```shell

```

### Relevant log output

```shell

```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.