cloudnative-pg / cloudnative-pg/plugin-barman-cloud
Former primary stuck at 1/2 after failover due to barman-cloud-check-wal-archive "Expected empty archive"
- Langage dominant
- Go
- Étoiles
- 191
- Forks
- 72
- Merge moyen
- 1 j 16 h
- PR mergées (30 j)
- 18
Description
### Description
After a failover/switchover on a running cluster (bootstrapped with `initdb`, **not** a recovery/restore), the `plugin-barman-cloud` sidecar on replica pods keeps failing with `"WAL archive check failed: Expected empty archive"`. This prevents the replica from becoming fully ready (1/2 containers).
The root cause is that `barman-cloud-check-wal-archive` is executed on **every WAL Archive gRPC call**, including on **replica pods**, and it fails because the S3 bucket already contains WAL files from previous timelines (which is expected after a switchover).
### Environment
- **CloudNativePG operator**: v1.28.1
- **plugin-barman-cloud**: v0.11.0
- **PostgreSQL**: 18.3 (`ghcr.io/cloudnative-pg/postgresql:18.3`)
- **Kubernetes**: v1.31
- **Object storage**: Ceph RGW (S3-compatible, via Rook)
### Cluster configuration
The cluster is bootstrapped with `initdb` (no recovery, no externalClusters):
```yaml
spec:
instances: 3
bootstrap:
initdb:
database: system
encoding: UTF8
owner: app
plugins:
- name: barman-cloud.cloudnative-pg.io
enabled: true
isWALArchiver: true
parameters:
barmanObjectName: my-object-store
```
### Steps to reproduce
1. Create a 3-instance CNPG cluster with `initdb` bootstrap and `plugin-barman-cloud` with `isWALArchiver: true`
2. Wait for all 3 pods to be 2/2 Running
3. A failover or switchover occurs (automatic or manual), changing the timeline (e.g., timeline 1 → 2 → 3)
4. After the failover, one or more replica pods get stuck at 1/2 — the `plugin-barman-cloud` sidecar blocks WAL archiving
### What happens
After the switchover, the former primary is demoted to replica. The instance manager detects leftover WAL files and explicitly triggers archiving on the demoted pod:
```
{"msg":"Detected ready WAL files in a former primary, triggering WAL archiving"}
```
This causes the `plugin-barman-cloud` sidecar to attempt WAL archiving. However, the S3 bucket legitimately contains WAL files from previous timelines (archived by this same pod when it was primary). The plugin executes `barman-cloud-check-wal-archive`, finds the bucket is not empty, and fails:
```
barman-cloud-check-wal-archive: ERROR: WAL archive check failed for server : Expected empty archive
```
plugin-barman-cloud sidecar log (replica pod):
```
{"level":"info","msg":"barman-cloud-check-wal-archive checking the first wal"}
{"level":"info","logger":"barman-cloud-check-wal-archive","msg":"ERROR: WAL archive check failed for server : Expected empty archive","pipe":"stderr"}
{"level":"error","msg":"Error invoking barman-cloud-check-wal-archive",
"options":["--endpoint-url","http://","--cloud-provider","aws-s3","s3://",""],
"exitCode":-1,"error":"exit status 1"}
```
### Expected behavior
All the cluster pods are correctly running after a switchover/failover.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Reproduisez le problème avec un cluster initdb à trois instances, avec le plugin plugin-barman-cloud et isWALArchiver activés, puis suivez l’appel WAL Archive gRPC et le comportement de barman-cloud-check-wal-archive sur l’ancien Primary. Vérifiez le chemin de Failover dans lequel un pod rétrogradé déclenche l’archivage malgré la présence de fichiers WAL provenant de timelines antérieures. Le travail est terminé lorsque tous les pods reviennent à 2/2 après un Switchover ou un Failover sans l’erreur Expected empty archive.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- go, kubernetes, postgresql
- Domaine
- databases, devops
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Active
- Clarté
- Plutôt claire
- Accessibilité débutants
- 48/100