[Bug]: VolumeSnapshot backup stopped working after upgrade to 1.28.x

### Is there an existing issue already for this bug?

- [x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.

### I have read the troubleshooting guide

- [x] I have read the troubleshooting guide and I think this is a new bug.

### I am running a supported version of CloudNativePG

- [x] I have read the troubleshooting guide and I think this is a new bug.

### Contact Details

p.krupa@avsystem.com

### Version

1.28 (latest patch)

### What version of Kubernetes are you using?

1.34

### What is your Kubernetes environment?

Cloud: Google GKE

### How did you install the operator?

Helm

### What happened?

After upgrading cnpg to 1.28.0 (and also to 1.28.1 during debug) my scheduledbackups based on volumeSnapshot stopped working. Downgrade to 1.27.2 resumed proper backups.

Inside controller there was only logs about backup starting and that a pod from which backup should be performed was found. Backup object itself stayed in `starting` phase for prolonged time and then logged this after some time:

```
'can''t execute snapshot backup: encountered a body error while preparing,
code: ''BACKUP_STATUS_CONTAINS_ERROR'', message: error while pinging: driver:
bad connection; sql: connection is already closed'
```

### Cluster resource

```shell
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
annotations:
meta.helm.sh/release-name: postgres-cluster
meta.helm.sh/release-namespace: iam-db
creationTimestamp: "2026-01-20T09:53:20Z"
generation: 4
labels:
app.kubernetes.io/managed-by: Helm
name: postgres
namespace: iam-db
resourceVersion: "1770374988054223024"
uid: 04f2c9af-5739-4d86-9224-9dd050d40659
spec:
affinity:
enablePodAntiAffinity: true
nodeSelector:
kubernetes.io/arch: amd64
podAntiAffinityType: preferred
topologyKey: kubernetes.io/hostname
backup:
target: prefer-standby
volumeSnapshot:
className: default
labels:
cleanup.kyverno.io/ttl: 7d
online: true
onlineConfiguration:
immediateCheckpoint: false
waitForArchive: true
snapshotOwnerReference: none
bootstrap:
initdb:
database: iam
encoding: UTF8
localeCType: C
localeCollate: C
owner: iam-user
secret:
name: postgres-user-pass
enablePDB: true
enableSuperuserAccess: false
failoverDelay: 0
imageCatalogRef:
apiGroup: postgresql.cnpg.io
kind: ClusterImageCatalog
major: 16
name: postgresql
inheritedMetadata:
annotations: {}
labels: {}
instances: 3
logLevel: debug
maxSyncReplicas: 0
minSyncReplicas: 0
monitoring:
customQueriesConfigMap:
- key: queries
name: cnpg-default-monitoring
disableDefaultQueries: false
enablePodMonitor: true
postgresGID: 26
postgresUID: 26
postgresql:
parameters:
archive_mode: "on"
archive_timeout: 5min
dynamic_shared_memory_type: posix
full_page_writes: "on"
huge_pages: "off"
log_destination: csvlog
log_directory: /controller/log
log_filename: postgres
log_rotation_age: "0"
log_rotation_size: "0"
log_truncate_on_rotation: "false"
logging_collector: "on"
max_connections: "200"
max_parallel_workers: "32"
max_replication_slots: "32"
max_slot_wal_keep_size: 8GB
max_worker_processes: "32"
shared_buffers: 128MB
shared_memory_type: mmap
shared_preload_libraries: ""
ssl_max_protocol_version: TLSv1.3
ssl_min_protocol_version: TLSv1.3
wal_keep_size: 512MB
wal_level: logical
wal_log_hints: "on"
wal_receiver_timeout: 5s
wal_sender_timeout: 5s
syncReplicaElectionConstraint:
enabled: false
primaryUpdateMethod: switchover
primaryUpdateStrategy: unsupervised
probes:
liveness:
isolationCheck:
connectionTimeout: 1000
enabled: true
requestTimeout: 1000
replicationSlots:
highAvailability:
enabled: true
slotPrefix: _cnpg_
synchronizeReplicas:
enabled: true
updateInterval: 30
resources:
limits:
cpu: "1"
memory: 4Gi
requests:
cpu: 250m
memory: 512Mi
smartShutdownTimeout: 180
startDelay: 300
stopDelay: 300
storage:
resizeInUseVolumes: true
size: 80Gi
storageClass: avs-re1-retain
superuserSecret:
name: postgres-admin
switchoverDelay: 600
status:
availableArchitectures:
- goArch: amd64
hash: 735bea2e670f099fa1fdeb8c522e11491eed66e9fe4da0b7db62225fe16125d9
- goArch: arm64
hash: dc260566fd1fd0a0083ff5caf4d8073d896d15453b8fd9abce7fc438a7b8ecf3
certificates:
clientCASecret: postgres-ca
expirations:
postgres-ca: 2026-04-20 09:48:20 +0000 UTC
postgres-replication: 2026-04-20 09:48:20 +0000 UTC
postgres-server: 2026-04-20 09:48:20 +0000 UTC
replicationTLSSecret: postgres-replication
serverAltDNSNames:
- postgres-rw
- postgres-rw.iam-db
- postgres-rw.iam-db.svc
- postgres-rw.iam-db.svc.cluster.local
- postgres-r
- postgres-r.iam-db
- postgres-r.iam-db.svc
- postgres-r.iam-db.svc.cluster.local
- postgres-ro
- postgres-ro.iam-db
- postgres-ro.iam-db.svc
- postgres-ro.iam-db.svc.cluster.local
serverCASecret: postgres-ca
serverTLSSecret: postgres-server
cloudNativePGCommitHash: 8f4702a29
cloudNativePGOperatorHash: 735bea2e670f099fa1fdeb8c522e11491eed66e9fe4da0b7db62225fe16125d9
conditions:
- lastTransitionTime: "2026-01-20T10:05:42Z"
message: A single, unique system ID was found across reporting instances.
reason: Unique
status: "True"
type: ConsistentSystemID
- lastTransitionTime: "2026-02-06T10:48:59Z"
message: Cluster is Ready
reason: ClusterIsReady
status: "True"
type: Ready
- lastTransitionTime: "2026-02-06T10:48:25Z"
message: Continuous archiving is working
reason: ContinuousArchivingSuccess
status: "True"
type: ContinuousArchiving
- lastTransitionTime: "2026-02-06T10:49:47Z"
message: Backup was successful
reason: LastBackupSucceeded
status: "True"
type: LastBackupSucceeded
configMapResourceVersion:
metrics:
cnpg-default-monitoring: "1768902801144751012"
currentPrimary: postgres-2
currentPrimaryTimestamp: "2026-02-06T10:48:24.505076Z"
firstRecoverabilityPoint: "2026-02-06T10:49:47Z"
firstRecoverabilityPointByMethod:
volumeSnapshot: "2026-02-06T10:49:47Z"
healthyPVC:
- postgres-1
- postgres-2
- postgres-4
image: registry.example.com/postgresql-16:2025.3.4
instanceNames:
- postgres-1
- postgres-2
- postgres-4
instances: 3
instancesReportedState:
postgres-1:
ip: 100.65.1.162
isPrimary: false
timeLineID: 8
postgres-2:
ip: 100.65.2.29
isPrimary: true
timeLineID: 8
postgres-4:
ip: 100.65.0.146
isPrimary: false
timeLineID: 7
instancesStatus:
healthy:
- postgres-1
- postgres-2
- postgres-4
lastFailedBackup: "2026-02-06T10:03:17Z"
lastSuccessfulBackup: "2026-02-06T10:49:47Z"
lastSuccessfulBackupByMethod:
volumeSnapshot: "2026-02-06T10:49:47Z"
latestGeneratedNode: 4
managedRolesStatus: {}
pgDataImageInfo:
image: registry.example.com/postgresql-16:2025.3.4
majorVersion: 16
phase: Cluster in healthy state
poolerIntegrations:
pgBouncerIntegration: {}
pvcCount: 3
readService: postgres-r
readyInstances: 3
secretsResourceVersion:
applicationSecretVersion: "1768902800742047003"
clientCaSecretVersion: "1768902800445791016"
replicationSecretVersion: "1768902800532895005"
serverCaSecretVersion: "1768902800445791016"
serverSecretVersion: "1768902800495247002"
switchReplicaClusterStatus: {}
systemID: "7597357323449516051"
targetPrimary: postgres-2
targetPrimaryTimestamp: "2026-02-06T10:48:22.712936Z"
timelineID: 8
topology:
instances:
postgres-1: {}
postgres-2: {}
postgres-4: {}
nodesUsed: 3
successfullyExtracted: true
writeService: postgres-rw
```

### Relevant log output

```shell
{"level":"debug","ts":"2026-02-06T11:25:48.589302271Z","msg":"reconciling object `iam-db/postgres-20260206122402`","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122402","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122402","reconcileID":"36aa9d1e-4be1-45fa-8c1d-1c19341b4718","caller":"internal/controller/backup_controller.go:120"}
{"level":"info","ts":"2026-02-06T11:25:52.440056415Z","logger":"backup-resource","msg":"Defaulting for Backup","version":"v1","name":"postgres-20260206122551","namespace":"iam-db"}
{"level":"info","ts":"2026-02-06T11:25:52.448935389Z","logger":"backup-resource","msg":"Validation for Backup upon creation","version":"v1","name":"postgres-20260206122551","namespace":"iam-db"}
{"level":"debug","ts":"2026-02-06T11:25:52.478215191Z","msg":"reconciling object `iam-db/postgres-20260206122551`","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"65026f29-338f-4c91-a3e1-8948c357431e","caller":"internal/controller/backup_controller.go:120"}
{"level":"debug","ts":"2026-02-06T11:25:52.478338364Z","msg":"Found cluster for backup","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"65026f29-338f-4c91-a3e1-8948c357431e","caller":"internal/controller/backup_controller.go:168","cluster":"postgres"}
{"level":"debug","ts":"2026-02-06T11:25:52.503998482Z","msg":"Standby Instance is elected as backup target","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"65026f29-338f-4c91-a3e1-8948c357431e","caller":"internal/controller/backup_controller.go:848","instance":"postgres-2"}
{"level":"debug","ts":"2026-02-06T11:25:52.562990799Z","msg":"Found pod for backup","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"65026f29-338f-4c91-a3e1-8948c357431e","caller":"internal/controller/backup_controller.go:728","pod":"postgres-2"}
{"level":"debug","ts":"2026-02-06T11:25:52.614493693Z","msg":"reconciling object `iam-db/postgres-20260206122551`","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"323e894c-f87d-4051-b31d-25e9f43336dd","caller":"internal/controller/backup_controller.go:120"}
{"level":"debug","ts":"2026-02-06T11:25:52.614606166Z","msg":"Found cluster for backup","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"323e894c-f87d-4051-b31d-25e9f43336dd","caller":"internal/controller/backup_controller.go:168","cluster":"postgres"}
{"level":"info","ts":"2026-02-06T11:25:52.614767372Z","msg":"Backup is already running on","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"323e894c-f87d-4051-b31d-25e9f43336dd","cluster":"postgres","pod":"postgres-2","startedAt":"2026-02-06 11:25:52 +0000 UTC"}
{"level":"debug","ts":"2026-02-06T11:25:52.614823868Z","msg":"Target pod is healthy for backup","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"323e894c-f87d-4051-b31d-25e9f43336dd","caller":"internal/controller/backup_controller.go:1036","podName":"postgres-2","backupName":"postgres-20260206122551"}
{"level":"info","ts":"2026-02-06T11:25:52.615043831Z","msg":"found a previously elected pod, reusing it","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"323e894c-f87d-4051-b31d-25e9f43336dd","targetPodName":"postgres-2"}
{"level":"debug","ts":"2026-02-06T11:25:57.615551564Z","msg":"reconciling object `iam-db/postgres-20260206122551`","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"f64842d0-bee1-485e-9084-ef177f858796","caller":"internal/controller/backup_controller.go:120"}
{"level":"debug","ts":"2026-02-06T11:25:57.615635491Z","msg":"Found cluster for backup","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"f64842d0-bee1-485e-9084-ef177f858796","caller":"internal/controller/backup_controller.go:168","cluster":"postgres"}
{"level":"info","ts":"2026-02-06T11:25:57.615808736Z","msg":"Backup is already running on","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"f64842d0-bee1-485e-9084-ef177f858796","cluster":"postgres","pod":"postgres-2","startedAt":"2026-02-06 11:25:52 +0000 UTC"}
{"level":"debug","ts":"2026-02-06T11:25:57.61586101Z","msg":"Target pod is healthy for backup","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"f64842d0-bee1-485e-9084-ef177f858796","caller":"internal/controller/backup_controller.go:1036","podName":"postgres-2","backupName":"postgres-20260206122551"}
{"level":"info","ts":"2026-02-06T11:25:57.616099955Z","msg":"found a previously elected pod, reusing it","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"postgres-20260206122551","namespace":"iam-db"},"namespace":"iam-db","name":"postgres-20260206122551","reconcileID":"f64842d0-bee1-485e-9084-ef177f858796","targetPodName":"postgres-2"}
```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct