cloudnative-pg / cloudnative-pg/plugin-barman-cloud

Former primary stuck at 1/2 after failover due to barman-cloud-check-wal-archive "Expected empty archive"

Ouverte
#828 7 commentaires 7 réactions 0 personnes assignées Voir sur GitHub
bug
Langage dominant
Go
Étoiles
191
Forks
72
Merge moyen
1 j 16 h
PR mergées (30 j)
18

Description

### Description

After a failover/switchover on a running cluster (bootstrapped with `initdb`, **not** a recovery/restore), the `plugin-barman-cloud` sidecar on replica pods keeps failing with `"WAL archive check failed: Expected empty archive"`. This prevents the replica from becoming fully ready (1/2 containers).

The root cause is that `barman-cloud-check-wal-archive` is executed on **every WAL Archive gRPC call**, including on **replica pods**, and it fails because the S3 bucket already contains WAL files from previous timelines (which is expected after a switchover).

### Environment

- **CloudNativePG operator**: v1.28.1
- **plugin-barman-cloud**: v0.11.0
- **PostgreSQL**: 18.3 (`ghcr.io/cloudnative-pg/postgresql:18.3`)
- **Kubernetes**: v1.31
- **Object storage**: Ceph RGW (S3-compatible, via Rook)

### Cluster configuration

The cluster is bootstrapped with `initdb` (no recovery, no externalClusters):

```yaml
spec:
instances: 3
bootstrap:
initdb:
database: system
encoding: UTF8
owner: app
plugins:
- name: barman-cloud.cloudnative-pg.io
enabled: true
isWALArchiver: true
parameters:
barmanObjectName: my-object-store
```

### Steps to reproduce

1. Create a 3-instance CNPG cluster with `initdb` bootstrap and `plugin-barman-cloud` with `isWALArchiver: true`
2. Wait for all 3 pods to be 2/2 Running
3. A failover or switchover occurs (automatic or manual), changing the timeline (e.g., timeline 1 → 2 → 3)
4. After the failover, one or more replica pods get stuck at 1/2 — the `plugin-barman-cloud` sidecar blocks WAL archiving

### What happens

After the switchover, the former primary is demoted to replica. The instance manager detects leftover WAL files and explicitly triggers archiving on the demoted pod:

```
{"msg":"Detected ready WAL files in a former primary, triggering WAL archiving"}
```

This causes the `plugin-barman-cloud` sidecar to attempt WAL archiving. However, the S3 bucket legitimately contains WAL files from previous timelines (archived by this same pod when it was primary). The plugin executes `barman-cloud-check-wal-archive`, finds the bucket is not empty, and fails:

```
barman-cloud-check-wal-archive: ERROR: WAL archive check failed for server : Expected empty archive
```

plugin-barman-cloud sidecar log (replica pod):

```
{"level":"info","msg":"barman-cloud-check-wal-archive checking the first wal"}
{"level":"info","logger":"barman-cloud-check-wal-archive","msg":"ERROR: WAL archive check failed for server : Expected empty archive","pipe":"stderr"}
{"level":"error","msg":"Error invoking barman-cloud-check-wal-archive",
"options":["--endpoint-url","http://","--cloud-provider","aws-s3","s3://",""],
"exitCode":-1,"error":"exit status 1"}
```

### Expected behavior
All the cluster pods are correctly running after a switchover/failover.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Reproduisez le problème avec un cluster initdb à trois instances, avec le plugin plugin-barman-cloud et isWALArchiver activés, puis suivez l’appel WAL Archive gRPC et le comportement de barman-cloud-check-wal-archive sur l’ancien Primary. Vérifiez le chemin de Failover dans lequel un pod rétrogradé déclenche l’archivage malgré la présence de fichiers WAL provenant de timelines antérieures. Le travail est terminé lorsque tous les pods reviennent à 2/2 après un Switchover ou un Failover sans l’erreur Expected empty archive.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
go, kubernetes, postgresql
Domaine
databases, devops
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Active
Clarté
Plutôt claire
Accessibilité débutants
48/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.