cloudnative-pg / cloudnative-pg/plugin-barman-cloud

Primary Pod takes ~3 minutes to terminate when the barman-cloud sidecar is enabled (5-10s without it)

Aperta
#1,039 1 commento 1 reazione 0 assegnatari Vedi su GitHub
bug
Lingua principale
Go
Stelle
191
Fork
72
Merge medio
1g 16h
PR unite (30g)
18

Descrizione

### What happened

On a single-instance CNPG `Cluster`, any change that rolls the primary takes **~3 minutes** to terminate the old Pod when the barman-cloud sidecar is enabled. With the plugin removed from the same cluster, the same roll terminates in **5–10 seconds**.

The delay is spent with the Pod in `Terminating` (`deletionTimestamp` set) while the containers are still reported running. PostgreSQL has already stopped accepting connections; the Pod simply does not go away.

### Measurements

Five consecutive primary rolls on one cluster, alternating the plugin on and off. Time is from `metadata.deletionTimestamp` being set to the Pod reaching a terminal state, sampled once per second:

| roll | trigger | sidecar present | termination |
|---|---|---|---|
| 1 | remove `spec.plugins` | **yes** | **3m10s** |
| 2 | add `spec.plugins` | no | 10s |
| 3 | `nodeAffinity` change | **yes** | **3m08s** |
| 4 | remove `spec.plugins` | **yes** | **3m03s** |
| 5 | add `spec.plugins` | no | 5s |

(In rolls 1, 3 and 4 the Pod being deleted carried `plugin-barman-cloud` as a native sidecar init container; in rolls 2 and 5 it did not, because the plugin had just been removed. Correlation is 5/5.)

The Pod's `initContainerStatuses` show `plugin-barman-cloud` in `running` for the whole window, only reaching `terminated` right at the end, immediately after which the Pod goes `Succeeded` and a replacement is created.

### What I ruled out

- **PostgreSQL smart shutdown.** This was my first assumption and it is wrong. The control is rolls 2 and 5: same cluster, same database, same `stopDelay`/`smartShutdownTimeout` (defaults), plugin absent → 5–10s. Additionally these runs had **no client sessions at all** (the only application using the database was scaled to zero), so there was nothing for a smart shutdown to wait on.
- **Hitting a grace deadline.** `terminationGracePeriodSeconds` is 1800 (CNPG's `stopDelay`), so ~3 minutes is not a deadline being reached — something completes at ~3 min.
- **Storage.** Volume attach/detach in these runs completes in ~6–7s and happens after the old Pod is gone.

### Why it matters

For a single-instance cluster the primary roll *is* the outage: nothing serves queries until the replacement is ready. The sidecar turns what should be a ~10 second interruption into ~3 minutes, on every roll — configuration changes, operator upgrades, PostgreSQL minor upgrades, or enabling the plugin itself.

It also widens the window for other failure modes. We spent some time chasing a recreate loop on a memory-pressured node; a 3-minute roll window instead of a 10-second one made that considerably easier to hit.

### Environment

- plugin-barman-cloud **v0.13.0** (sidecar image `plugin-barman-cloud-sidecar:v0.13.0`)
- CloudNativePG operator **1.30.0**
- PostgreSQL 18.3 (`postgresql:18.3-minimal-trixie`)
- k3s, Longhorn RWO storage, `instances: 1`, `primaryUpdateMethod: restart`
- ObjectStore backed by S3-compatible storage; WAL archiving enabled (`isWALArchiver: true`)

### Note on v0.14.0

These measurements are from v0.13.0. v0.14.0 was released while I was writing this up; I read its changelog and see no change related to sidecar shutdown, so I believe this still applies — but I have not re-measured on it. Happy to do so, and happy to gather any additional detail (sidecar logs at debug, goroutine dump on SIGTERM) if that would help.

### Expected behaviour

Once the main containers have stopped, the sidecar should exit promptly on SIGTERM rather than keeping the Pod alive for ~3 minutes.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia riproducendo la terminazione del Pod primario con plugin-barman-cloud v0.13.0, confrontando i rollout con e senza il sidecar. Ispeziona il comportamento di SIGTERM e dello shutdown del sidecar, usando i log di debug o un dump delle goroutine se necessario. Il lavoro è completato quando il sidecar termina rapidamente dopo l’arresto dei container principali e il Pod torna alla normale breve finestra di terminazione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
go, kubernetes, postgresql
Ambito
databases, infrastructure
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
48/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.