cloudnative-pg / cloudnative-pg/plugin-barman-cloud

Primary Pod takes ~3 minutes to terminate when the barman-cloud sidecar is enabled (5-10s without it)

Open
#1,039 1 comment 1 reaction 0 assignees View on GitHub
bug
Dominant language
Go
Stars
191
Forks
72
Avg merge
2d 21h
Merged PRs (30d)
21

Description

### What happened

On a single-instance CNPG `Cluster`, any change that rolls the primary takes **~3 minutes** to terminate the old Pod when the barman-cloud sidecar is enabled. With the plugin removed from the same cluster, the same roll terminates in **5–10 seconds**.

The delay is spent with the Pod in `Terminating` (`deletionTimestamp` set) while the containers are still reported running. PostgreSQL has already stopped accepting connections; the Pod simply does not go away.

### Measurements

Five consecutive primary rolls on one cluster, alternating the plugin on and off. Time is from `metadata.deletionTimestamp` being set to the Pod reaching a terminal state, sampled once per second:

| roll | trigger | sidecar present | termination |
|---|---|---|---|
| 1 | remove `spec.plugins` | **yes** | **3m10s** |
| 2 | add `spec.plugins` | no | 10s |
| 3 | `nodeAffinity` change | **yes** | **3m08s** |
| 4 | remove `spec.plugins` | **yes** | **3m03s** |
| 5 | add `spec.plugins` | no | 5s |

(In rolls 1, 3 and 4 the Pod being deleted carried `plugin-barman-cloud` as a native sidecar init container; in rolls 2 and 5 it did not, because the plugin had just been removed. Correlation is 5/5.)

The Pod's `initContainerStatuses` show `plugin-barman-cloud` in `running` for the whole window, only reaching `terminated` right at the end, immediately after which the Pod goes `Succeeded` and a replacement is created.

### What I ruled out

- **PostgreSQL smart shutdown.** This was my first assumption and it is wrong. The control is rolls 2 and 5: same cluster, same database, same `stopDelay`/`smartShutdownTimeout` (defaults), plugin absent → 5–10s. Additionally these runs had **no client sessions at all** (the only application using the database was scaled to zero), so there was nothing for a smart shutdown to wait on.
- **Hitting a grace deadline.** `terminationGracePeriodSeconds` is 1800 (CNPG's `stopDelay`), so ~3 minutes is not a deadline being reached — something completes at ~3 min.
- **Storage.** Volume attach/detach in these runs completes in ~6–7s and happens after the old Pod is gone.

### Why it matters

For a single-instance cluster the primary roll *is* the outage: nothing serves queries until the replacement is ready. The sidecar turns what should be a ~10 second interruption into ~3 minutes, on every roll — configuration changes, operator upgrades, PostgreSQL minor upgrades, or enabling the plugin itself.

It also widens the window for other failure modes. We spent some time chasing a recreate loop on a memory-pressured node; a 3-minute roll window instead of a 10-second one made that considerably easier to hit.

### Environment

- plugin-barman-cloud **v0.13.0** (sidecar image `plugin-barman-cloud-sidecar:v0.13.0`)
- CloudNativePG operator **1.30.0**
- PostgreSQL 18.3 (`postgresql:18.3-minimal-trixie`)
- k3s, Longhorn RWO storage, `instances: 1`, `primaryUpdateMethod: restart`
- ObjectStore backed by S3-compatible storage; WAL archiving enabled (`isWALArchiver: true`)

### Note on v0.14.0

These measurements are from v0.13.0. v0.14.0 was released while I was writing this up; I read its changelog and see no change related to sidecar shutdown, so I believe this still applies — but I have not re-measured on it. Happy to do so, and happy to gather any additional detail (sidecar logs at debug, goroutine dump on SIGTERM) if that would help.

### Expected behaviour

Once the main containers have stopped, the sidecar should exit promptly on SIGTERM rather than keeping the Pod alive for ~3 minutes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.