cloudnative-pg / cloudnative-pg/plugin-barman-cloud

Failure to restore from backup due to missing WAL file

Open
#618 1 comment 2 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
191
Forks
72
Avg merge
2d 21h
Merged PRs (30d)
21

Description

I created a backup, when it was checked as finished I tried deleting and restoring the cluster but it got stuck in a state trying to find a missing WAL file.

I couldn't find a way to load only from the base and ignore the WAL files completely either.

This is how my files were set up

Only WAL present in AWS was `/wals/0000000600000000/0000000600000000000000AA.gz`

backup.info had
```
backup_label='START WAL LOCATION: 0/AB000028 (file 0000000600000000000000AB)\nCHECKPOINT LOCATION: 0/AB000028\nBACKUP METHOD: streamed\nBACKUP FROM: standby\nSTART TIME: 2025-10-24 02:26:08 UTC\nLABEL: Barman backup cloud 20251024T022608\nSTART TIMELINE: 6\n'
backup_name=backup-20251024022604
begin_offset=40
begin_time=2025-10-24 02:26:08.217166+00:00
begin_wal=0000000600000000000000AB
begin_xlog=0/AB000028
children_backup_ids=None
cluster_size=48071936
compression=None
config_file=/var/lib/postgresql/data/pgdata/postgresql.conf
copy_stats={'total_time': 3.17413, 'number_of_workers': 2, 'analysis_time': 0, 'analysis_time_per_item': {'data': 0}, 'copy_time_per_item': {'data': 1.796744}, 'serialized_copy_time_per_item': {'data': 1.924254}, 'copy_time': 1.796744, 'serialized_copy_time': 1.924254}
data_checksums=off
deduplicated_size=None
encryption=None
end_offset=416
end_time=2025-10-24 02:26:11.034605+00:00
end_wal=0000000600000000000000AB
end_xlog=0/AB0001A0
error=None
hba_file=/var/lib/postgresql/data/pgdata/pg_hba.conf
ident_file=/var/lib/postgresql/data/pgdata/pg_ident.conf
included_files=['/var/lib/postgresql/data/pgdata/custom.conf', '/var/lib/postgresql/data/pgdata/override.conf']
mode=None
parent_backup_id=None
pgdata=/var/lib/postgresql/data/pgdata
server_name=cloud
size=None
status=DONE
summarize_wal=off
systemid=7564138978558590997
tablespaces=None
timeline=6
version=170006
xlog_segment_size=16777216
```

inside `data.tar` the `backup_label` had:

```
START WAL LOCATION: 0/AB000028 (file 0000000600000000000000AB)
CHECKPOINT LOCATION: 0/AB000028
BACKUP METHOD: streamed
BACKUP FROM: standby
START TIME: 2025-10-24 02:26:08 UTC
LABEL: Barman backup cloud 20251024T022608
START TIMELINE: 6
```

Pod log
```
{"level":"info","ts":"2025-10-24T02:46:04.754995341Z","msg":"Starting webserver","logging_pod":"postgres-cluster-1-full-recovery","address":"localhost:8010","hasTLS":false}
{"level":"info","ts":"2025-10-24T02:46:04.860377215Z","msg":"Restore through plugin detected, proceeding...","logging_pod":"postgres-cluster-1-full-recovery"}
{"level":"error","ts":"2025-10-24T02:46:12.557539207Z","msg":"Error while restoring a backup","logging_pod":"postgres-cluster-1-full-recovery","error":"rpc error: code = Unknown desc = encountered an error while checking the presence of first needed WAL in the archive: object storage or file not found 0000000600000000000000AB: WAL not found","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.1/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/restore.restoreSubCommand\n\tinternal/cmd/manager/instance/restore/restore.go:79\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/restore.(*restoreRunnable).Start\n\tinternal/cmd/manager/instance/restore/restore.go:62\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.3/pkg/manager/runnable_group.go:260"}
{"level":"info","ts":"2025-10-24T02:46:12.569395329Z","msg":"Stopping and waiting for non leader election runnables"}
{"level":"info","ts":"2025-10-24T02:46:12.569421782Z","msg":"Stopping and waiting for leader election runnables"}
{"level":"info","ts":"2025-10-24T02:46:12.574760522Z","msg":"Webserver exited","logging_pod":"postgres-cluster-1-full-recovery","address":"localhost:8010"}
{"level":"info","ts":"2025-10-24T02:46:12.57576866Z","msg":"Stopping and waiting for caches"}
{"level":"info","ts":"2025-10-24T02:46:12.574699681Z","msg":"Stopping and waiting for warmup runnables"}
{"level":"info","ts":"2025-10-24T02:46:12.578650618Z","msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":"2025-10-24T02:46:12.578679668Z","msg":"Stopping and waiting for HTTP servers"}
{"level":"info","ts":"2025-10-24T02:46:12.578689889Z","msg":"Wait completed, proceeding to shutdown the manager"}
{"level":"error","ts":"2025-10-24T02:46:12.578836461Z","msg":"restore error","logging_pod":"postgres-cluster-1-full-recovery","error":"while restoring cluster: rpc error: code = Unknown desc = encountered an error while checking the presence of first needed WAL in the archive: object storage or file not found 0000000600000000000000AB: WAL not found","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.1/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/restore.NewCmd.func1\n\tinternal/cmd/manager/instance/restore/cmd.go:101\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.3/x64/src/runtime/proc.go:285"}
```

My ObjectStore

```
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: postgres-backup-s3
namespace: superset
spec:
configuration:
destinationPath: s3://withub-backup-cloudnative-postgres/superset
s3Credentials:
inheritFromIAMRole: true
wal:
compression: gzip
retentionPolicy: 30d
status:
serverRecoveryWindow:
postgres-cluster-restored:
firstRecoverabilityPoint: '2025-10-24T02:18:01Z'
lastSuccessfulBackupTime: '2025-10-24T02:26:11Z'
```

My Backup CRD

```
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
labels:
cnpg.io/cluster: postgres-cluster
cnpg.io/immediateBackup: 'true'
cnpg.io/scheduled-backup: postgres-scheduled-backup
name: postgres-scheduled-backup-20251024021602
namespace: superset
spec:
cluster:
name: postgres-cluster
method: plugin
pluginConfiguration:
name: barman-cloud.cloudnative-pg.io
status:
backupId: 20251024T022608
backupName: backup-20251024022604
beginLSN: 0/AB000028
beginWal: 0000000600000000000000AB
endLSN: 0/AB0001A0
endWal: 0000000600000000000000AB
instanceID:
ContainerID: >-
containerd://2594945321f8c0aca511cf07ee38eea4e303253888ae84e1219f2b80dab5ef5c
podName: postgres-cluster-1
majorVersion: 17
method: plugin
online: true
phase: completed
pluginMetadata:
clusterUID: e6e34f8d-e925-437f-8e98-dd6425e10a03
displayName: BarmanCloudInstance
name: barman-cloud.cloudnative-pg.io
pluginName: barman-cloud.cloudnative-pg.io
timeline: '6'
version: 0.7.0
startedAt: '2025-10-24T02:26:08Z'
stoppedAt: '2025-10-24T02:26:11Z'
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.