cloudnative-pg / cloudnative-pg/plugin-barman-cloud
Failure to restore from backup due to missing WAL file
- Ngôn ngữ chính
- Go
- Star
- 191
- Fork
- 72
- Merge trung bình
- 1 ngày 16 giờ
- Pull request đã merge (30 ngày)
- 18
Mô tả
I created a backup, when it was checked as finished I tried deleting and restoring the cluster but it got stuck in a state trying to find a missing WAL file.
I couldn't find a way to load only from the base and ignore the WAL files completely either.
This is how my files were set up
Only WAL present in AWS was `/wals/0000000600000000/0000000600000000000000AA.gz`
backup.info had
```
backup_label='START WAL LOCATION: 0/AB000028 (file 0000000600000000000000AB)\nCHECKPOINT LOCATION: 0/AB000028\nBACKUP METHOD: streamed\nBACKUP FROM: standby\nSTART TIME: 2025-10-24 02:26:08 UTC\nLABEL: Barman backup cloud 20251024T022608\nSTART TIMELINE: 6\n'
backup_name=backup-20251024022604
begin_offset=40
begin_time=2025-10-24 02:26:08.217166+00:00
begin_wal=0000000600000000000000AB
begin_xlog=0/AB000028
children_backup_ids=None
cluster_size=48071936
compression=None
config_file=/var/lib/postgresql/data/pgdata/postgresql.conf
copy_stats={'total_time': 3.17413, 'number_of_workers': 2, 'analysis_time': 0, 'analysis_time_per_item': {'data': 0}, 'copy_time_per_item': {'data': 1.796744}, 'serialized_copy_time_per_item': {'data': 1.924254}, 'copy_time': 1.796744, 'serialized_copy_time': 1.924254}
data_checksums=off
deduplicated_size=None
encryption=None
end_offset=416
end_time=2025-10-24 02:26:11.034605+00:00
end_wal=0000000600000000000000AB
end_xlog=0/AB0001A0
error=None
hba_file=/var/lib/postgresql/data/pgdata/pg_hba.conf
ident_file=/var/lib/postgresql/data/pgdata/pg_ident.conf
included_files=['/var/lib/postgresql/data/pgdata/custom.conf', '/var/lib/postgresql/data/pgdata/override.conf']
mode=None
parent_backup_id=None
pgdata=/var/lib/postgresql/data/pgdata
server_name=cloud
size=None
status=DONE
summarize_wal=off
systemid=7564138978558590997
tablespaces=None
timeline=6
version=170006
xlog_segment_size=16777216
```
inside `data.tar` the `backup_label` had:
```
START WAL LOCATION: 0/AB000028 (file 0000000600000000000000AB)
CHECKPOINT LOCATION: 0/AB000028
BACKUP METHOD: streamed
BACKUP FROM: standby
START TIME: 2025-10-24 02:26:08 UTC
LABEL: Barman backup cloud 20251024T022608
START TIMELINE: 6
```
Pod log
```
{"level":"info","ts":"2025-10-24T02:46:04.754995341Z","msg":"Starting webserver","logging_pod":"postgres-cluster-1-full-recovery","address":"localhost:8010","hasTLS":false}
{"level":"info","ts":"2025-10-24T02:46:04.860377215Z","msg":"Restore through plugin detected, proceeding...","logging_pod":"postgres-cluster-1-full-recovery"}
{"level":"error","ts":"2025-10-24T02:46:12.557539207Z","msg":"Error while restoring a backup","logging_pod":"postgres-cluster-1-full-recovery","error":"rpc error: code = Unknown desc = encountered an error while checking the presence of first needed WAL in the archive: object storage or file not found 0000000600000000000000AB: WAL not found","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.1/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/restore.restoreSubCommand\n\tinternal/cmd/manager/instance/restore/restore.go:79\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/restore.(*restoreRunnable).Start\n\tinternal/cmd/manager/instance/restore/restore.go:62\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.3/pkg/manager/runnable_group.go:260"}
{"level":"info","ts":"2025-10-24T02:46:12.569395329Z","msg":"Stopping and waiting for non leader election runnables"}
{"level":"info","ts":"2025-10-24T02:46:12.569421782Z","msg":"Stopping and waiting for leader election runnables"}
{"level":"info","ts":"2025-10-24T02:46:12.574760522Z","msg":"Webserver exited","logging_pod":"postgres-cluster-1-full-recovery","address":"localhost:8010"}
{"level":"info","ts":"2025-10-24T02:46:12.57576866Z","msg":"Stopping and waiting for caches"}
{"level":"info","ts":"2025-10-24T02:46:12.574699681Z","msg":"Stopping and waiting for warmup runnables"}
{"level":"info","ts":"2025-10-24T02:46:12.578650618Z","msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":"2025-10-24T02:46:12.578679668Z","msg":"Stopping and waiting for HTTP servers"}
{"level":"info","ts":"2025-10-24T02:46:12.578689889Z","msg":"Wait completed, proceeding to shutdown the manager"}
{"level":"error","ts":"2025-10-24T02:46:12.578836461Z","msg":"restore error","logging_pod":"postgres-cluster-1-full-recovery","error":"while restoring cluster: rpc error: code = Unknown desc = encountered an error while checking the presence of first needed WAL in the archive: object storage or file not found 0000000600000000000000AB: WAL not found","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.1/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/restore.NewCmd.func1\n\tinternal/cmd/manager/instance/restore/cmd.go:101\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.3/x64/src/runtime/proc.go:285"}
```
My ObjectStore
```
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: postgres-backup-s3
namespace: superset
spec:
configuration:
destinationPath: s3://withub-backup-cloudnative-postgres/superset
s3Credentials:
inheritFromIAMRole: true
wal:
compression: gzip
retentionPolicy: 30d
status:
serverRecoveryWindow:
postgres-cluster-restored:
firstRecoverabilityPoint: '2025-10-24T02:18:01Z'
lastSuccessfulBackupTime: '2025-10-24T02:26:11Z'
```
My Backup CRD
```
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
labels:
cnpg.io/cluster: postgres-cluster
cnpg.io/immediateBackup: 'true'
cnpg.io/scheduled-backup: postgres-scheduled-backup
name: postgres-scheduled-backup-20251024021602
namespace: superset
spec:
cluster:
name: postgres-cluster
method: plugin
pluginConfiguration:
name: barman-cloud.cloudnative-pg.io
status:
backupId: 20251024T022608
backupName: backup-20251024022604
beginLSN: 0/AB000028
beginWal: 0000000600000000000000AB
endLSN: 0/AB0001A0
endWal: 0000000600000000000000AB
instanceID:
ContainerID: >-
containerd://2594945321f8c0aca511cf07ee38eea4e303253888ae84e1219f2b80dab5ef5c
podName: postgres-cluster-1
majorVersion: 17
method: plugin
online: true
phase: completed
pluginMetadata:
clusterUID: e6e34f8d-e925-437f-8e98-dd6425e10a03
displayName: BarmanCloudInstance
name: barman-cloud.cloudnative-pg.io
pluginName: barman-cloud.cloudnative-pg.io
timeline: '6'
version: 0.7.0
startedAt: '2025-10-24T02:26:08Z'
stoppedAt: '2025-10-24T02:26:11Z'
```
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu với internal/cmd/manager/instance/restore/restore.go tại phần kiểm tra sự hiện diện của WAL quanh dòng 79, sau đó lần theo đường dẫn restore của plugin bằng các log, metadata của backup và cấu hình ObjectStore được cung cấp. Điều tra lý do WAL đầu tiên bắt buộc bị coi là không tồn tại mặc dù backup đã hoàn tất, và xác minh kết quả bằng một reproduction của restore hoặc coverage hồi quy xác lập cách xử lý mong đợi.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- aws, go, postgresql
- Lĩnh vực
- cloud, databases
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 35/100