cloudnative-pg / cloudnative-pg/plugin-barman-cloud
Primary Pod takes ~3 minutes to terminate when the barman-cloud sidecar is enabled (5-10s without it)
- Lingua principale
- Go
- Stelle
- 191
- Fork
- 72
- Merge medio
- 1g 16h
- PR unite (30g)
- 18
Descrizione
### What happened
On a single-instance CNPG `Cluster`, any change that rolls the primary takes **~3 minutes** to terminate the old Pod when the barman-cloud sidecar is enabled. With the plugin removed from the same cluster, the same roll terminates in **5–10 seconds**.
The delay is spent with the Pod in `Terminating` (`deletionTimestamp` set) while the containers are still reported running. PostgreSQL has already stopped accepting connections; the Pod simply does not go away.
### Measurements
Five consecutive primary rolls on one cluster, alternating the plugin on and off. Time is from `metadata.deletionTimestamp` being set to the Pod reaching a terminal state, sampled once per second:
| roll | trigger | sidecar present | termination |
|---|---|---|---|
| 1 | remove `spec.plugins` | **yes** | **3m10s** |
| 2 | add `spec.plugins` | no | 10s |
| 3 | `nodeAffinity` change | **yes** | **3m08s** |
| 4 | remove `spec.plugins` | **yes** | **3m03s** |
| 5 | add `spec.plugins` | no | 5s |
(In rolls 1, 3 and 4 the Pod being deleted carried `plugin-barman-cloud` as a native sidecar init container; in rolls 2 and 5 it did not, because the plugin had just been removed. Correlation is 5/5.)
The Pod's `initContainerStatuses` show `plugin-barman-cloud` in `running` for the whole window, only reaching `terminated` right at the end, immediately after which the Pod goes `Succeeded` and a replacement is created.
### What I ruled out
- **PostgreSQL smart shutdown.** This was my first assumption and it is wrong. The control is rolls 2 and 5: same cluster, same database, same `stopDelay`/`smartShutdownTimeout` (defaults), plugin absent → 5–10s. Additionally these runs had **no client sessions at all** (the only application using the database was scaled to zero), so there was nothing for a smart shutdown to wait on.
- **Hitting a grace deadline.** `terminationGracePeriodSeconds` is 1800 (CNPG's `stopDelay`), so ~3 minutes is not a deadline being reached — something completes at ~3 min.
- **Storage.** Volume attach/detach in these runs completes in ~6–7s and happens after the old Pod is gone.
### Why it matters
For a single-instance cluster the primary roll *is* the outage: nothing serves queries until the replacement is ready. The sidecar turns what should be a ~10 second interruption into ~3 minutes, on every roll — configuration changes, operator upgrades, PostgreSQL minor upgrades, or enabling the plugin itself.
It also widens the window for other failure modes. We spent some time chasing a recreate loop on a memory-pressured node; a 3-minute roll window instead of a 10-second one made that considerably easier to hit.
### Environment
- plugin-barman-cloud **v0.13.0** (sidecar image `plugin-barman-cloud-sidecar:v0.13.0`)
- CloudNativePG operator **1.30.0**
- PostgreSQL 18.3 (`postgresql:18.3-minimal-trixie`)
- k3s, Longhorn RWO storage, `instances: 1`, `primaryUpdateMethod: restart`
- ObjectStore backed by S3-compatible storage; WAL archiving enabled (`isWALArchiver: true`)
### Note on v0.14.0
These measurements are from v0.13.0. v0.14.0 was released while I was writing this up; I read its changelog and see no change related to sidecar shutdown, so I believe this still applies — but I have not re-measured on it. Happy to do so, and happy to gather any additional detail (sidecar logs at debug, goroutine dump on SIGTERM) if that would help.
### Expected behaviour
Once the main containers have stopped, the sidecar should exit promptly on SIGTERM rather than keeping the Pod alive for ~3 minutes.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia riproducendo la terminazione del Pod primario con plugin-barman-cloud v0.13.0, confrontando i rollout con e senza il sidecar. Ispeziona il comportamento di SIGTERM e dello shutdown del sidecar, usando i log di debug o un dump delle goroutine se necessario. Il lavoro è completato quando il sidecar termina rapidamente dopo l’arresto dei container principali e il Pod torna alla normale breve finestra di terminazione.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- go, kubernetes, postgresql
- Ambito
- databases, infrastructure
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 48/100