cloudnative-pg / cloudnative-pg/plugin-barman-cloud

Primary Pod takes ~3 minutes to terminate when the barman-cloud sidecar is enabled (5-10s without it)

Offen
#1,039 1 Kommentar 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Go
Sterne
192
Forks
75
Ø Merge
1 T. 16 Std.
Gemergte PRs (30 T.)
18

Beschreibung

### What happened

On a single-instance CNPG `Cluster`, any change that rolls the primary takes **~3 minutes** to terminate the old Pod when the barman-cloud sidecar is enabled. With the plugin removed from the same cluster, the same roll terminates in **5–10 seconds**.

The delay is spent with the Pod in `Terminating` (`deletionTimestamp` set) while the containers are still reported running. PostgreSQL has already stopped accepting connections; the Pod simply does not go away.

### Measurements

Five consecutive primary rolls on one cluster, alternating the plugin on and off. Time is from `metadata.deletionTimestamp` being set to the Pod reaching a terminal state, sampled once per second:

| roll | trigger | sidecar present | termination |
|---|---|---|---|
| 1 | remove `spec.plugins` | **yes** | **3m10s** |
| 2 | add `spec.plugins` | no | 10s |
| 3 | `nodeAffinity` change | **yes** | **3m08s** |
| 4 | remove `spec.plugins` | **yes** | **3m03s** |
| 5 | add `spec.plugins` | no | 5s |

(In rolls 1, 3 and 4 the Pod being deleted carried `plugin-barman-cloud` as a native sidecar init container; in rolls 2 and 5 it did not, because the plugin had just been removed. Correlation is 5/5.)

The Pod's `initContainerStatuses` show `plugin-barman-cloud` in `running` for the whole window, only reaching `terminated` right at the end, immediately after which the Pod goes `Succeeded` and a replacement is created.

### What I ruled out

- **PostgreSQL smart shutdown.** This was my first assumption and it is wrong. The control is rolls 2 and 5: same cluster, same database, same `stopDelay`/`smartShutdownTimeout` (defaults), plugin absent → 5–10s. Additionally these runs had **no client sessions at all** (the only application using the database was scaled to zero), so there was nothing for a smart shutdown to wait on.
- **Hitting a grace deadline.** `terminationGracePeriodSeconds` is 1800 (CNPG's `stopDelay`), so ~3 minutes is not a deadline being reached — something completes at ~3 min.
- **Storage.** Volume attach/detach in these runs completes in ~6–7s and happens after the old Pod is gone.

### Why it matters

For a single-instance cluster the primary roll *is* the outage: nothing serves queries until the replacement is ready. The sidecar turns what should be a ~10 second interruption into ~3 minutes, on every roll — configuration changes, operator upgrades, PostgreSQL minor upgrades, or enabling the plugin itself.

It also widens the window for other failure modes. We spent some time chasing a recreate loop on a memory-pressured node; a 3-minute roll window instead of a 10-second one made that considerably easier to hit.

### Environment

- plugin-barman-cloud **v0.13.0** (sidecar image `plugin-barman-cloud-sidecar:v0.13.0`)
- CloudNativePG operator **1.30.0**
- PostgreSQL 18.3 (`postgresql:18.3-minimal-trixie`)
- k3s, Longhorn RWO storage, `instances: 1`, `primaryUpdateMethod: restart`
- ObjectStore backed by S3-compatible storage; WAL archiving enabled (`isWALArchiver: true`)

### Note on v0.14.0

These measurements are from v0.13.0. v0.14.0 was released while I was writing this up; I read its changelog and see no change related to sidecar shutdown, so I believe this still applies — but I have not re-measured on it. Happy to do so, and happy to gather any additional detail (sidecar logs at debug, goroutine dump on SIGTERM) if that would help.

### Expected behaviour

Once the main containers have stopped, the sidecar should exit promptly on SIGTERM rather than keeping the Pod alive for ~3 minutes.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne damit, die Beendigung des primären Pod mit plugin-barman-cloud v0.13.0 zu reproduzieren, und vergleiche die Rollouts mit und ohne Sidecar. Untersuche das SIGTERM- und Shutdown-Verhalten des Sidecars und verwende bei Bedarf Debug-Logs oder einen Goroutine-Dump. Die Aufgabe ist erledigt, wenn das Sidecar sofort beendet wird, nachdem die Hauptcontainer gestoppt wurden, und der Pod wieder in das normale kurze Beendigungszeitfenster zurückkehrt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
go, kubernetes, postgresql
Bereich
databases, infrastructure
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
48/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.