cloudnative-pg / cloudnative-pg/plugin-barman-cloud

Former primary stuck at 1/2 after failover due to barman-cloud-check-wal-archive "Expected empty archive"

Abierto
#828 7 comentarios 7 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Go
Estrellas
191
Forks
72
Merge medio
2 d 21 h
PR fusionados (30 d)
21

Descripción

### Description

After a failover/switchover on a running cluster (bootstrapped with `initdb`, **not** a recovery/restore), the `plugin-barman-cloud` sidecar on replica pods keeps failing with `"WAL archive check failed: Expected empty archive"`. This prevents the replica from becoming fully ready (1/2 containers).

The root cause is that `barman-cloud-check-wal-archive` is executed on **every WAL Archive gRPC call**, including on **replica pods**, and it fails because the S3 bucket already contains WAL files from previous timelines (which is expected after a switchover).

### Environment

- **CloudNativePG operator**: v1.28.1
- **plugin-barman-cloud**: v0.11.0
- **PostgreSQL**: 18.3 (`ghcr.io/cloudnative-pg/postgresql:18.3`)
- **Kubernetes**: v1.31
- **Object storage**: Ceph RGW (S3-compatible, via Rook)

### Cluster configuration

The cluster is bootstrapped with `initdb` (no recovery, no externalClusters):

```yaml
spec:
instances: 3
bootstrap:
initdb:
database: system
encoding: UTF8
owner: app
plugins:
- name: barman-cloud.cloudnative-pg.io
enabled: true
isWALArchiver: true
parameters:
barmanObjectName: my-object-store
```

### Steps to reproduce

1. Create a 3-instance CNPG cluster with `initdb` bootstrap and `plugin-barman-cloud` with `isWALArchiver: true`
2. Wait for all 3 pods to be 2/2 Running
3. A failover or switchover occurs (automatic or manual), changing the timeline (e.g., timeline 1 → 2 → 3)
4. After the failover, one or more replica pods get stuck at 1/2 — the `plugin-barman-cloud` sidecar blocks WAL archiving

### What happens

After the switchover, the former primary is demoted to replica. The instance manager detects leftover WAL files and explicitly triggers archiving on the demoted pod:

```
{"msg":"Detected ready WAL files in a former primary, triggering WAL archiving"}
```

This causes the `plugin-barman-cloud` sidecar to attempt WAL archiving. However, the S3 bucket legitimately contains WAL files from previous timelines (archived by this same pod when it was primary). The plugin executes `barman-cloud-check-wal-archive`, finds the bucket is not empty, and fails:

```
barman-cloud-check-wal-archive: ERROR: WAL archive check failed for server : Expected empty archive
```

plugin-barman-cloud sidecar log (replica pod):

```
{"level":"info","msg":"barman-cloud-check-wal-archive checking the first wal"}
{"level":"info","logger":"barman-cloud-check-wal-archive","msg":"ERROR: WAL archive check failed for server : Expected empty archive","pipe":"stderr"}
{"level":"error","msg":"Error invoking barman-cloud-check-wal-archive",
"options":["--endpoint-url","http://","--cloud-provider","aws-s3","s3://",""],
"exitCode":-1,"error":"exit status 1"}
```

### Expected behavior
All the cluster pods are correctly running after a switchover/failover.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Reproduzca el problema con un clúster initdb de tres instancias con el plugin plugin-barman-cloud y isWALArchiver habilitados; después, siga la llamada WAL Archive gRPC y el comportamiento de barman-cloud-check-wal-archive en el antiguo Primary. Verifique la ruta de Failover en la que un pod degradado activa el archivado a pesar de que ya existan archivos WAL de timelines anteriores. Se considera terminado cuando todos los pods vuelvan a 2/2 después de un Switchover o Failover sin el error Expected empty archive.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
go, kubernetes, postgresql
Área
databases, devops
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Activo
Claridad
Bastante claro
Aptitud para principiantes
48/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.