zalando / zalando/postgres-operator
Not restoring from backup because of amount of generated wals
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
Please, answer some short questions which should help us to understand your problem / question better?
- Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.6.0
- Where do you run it - cloud or metal? Kubernetes or OpenShift? [Bare Metal K8s]
- Are you running Postgres Operator in production? yes
- Type of issue? question
Please give me an advice. I get error during bootstrapping the replica from leader after I reinit the patroni in replica. It can not restore from WAL because of size of last backup. How can I solve this problem gracefully? WAL backups are stored in S3 storage. Here is my logs from replica:
2021-04-29 16:45:44,741 INFO: Lock owner: app-db-postgres-0; I am app-db-postgres-1
2021-04-29 16:45:44,741 INFO: bootstrap from leader 'app-db-postgres-0' in progress
pg_basebackup: error: COPY stream ended before last file was finished
pg_basebackup: removing data directory "/home/postgres/pgdata/pgroot/data"
2021-04-29 16:45:54,113 ERROR: Error creating replica using method basebackup_fast_xlog: /scripts/basebackup.sh exited with code=1
2021-04-29 16:45:54,113 ERROR: failed to bootstrap from leader 'app-db-postgres-0'
2021-04-29 16:45:54,113 INFO: Removing data directory: /home/postgres/pgdata/pgroot/data
2021-04-29 16:45:54,741 INFO: Lock owner: app-db-postgres-0; I am app-db-postgres-1
2021-04-29 16:45:54,761 INFO: trying to bootstrap from leader 'app-db-postgres-0'
Current cluster size: 72570816836
Wals generated since the last backup: 305608353752
not restoring from backup because of amount of generated wals exceeds 102400MB
2021-04-29 16:45:55,585 ERROR: Error creating replica using method wal_e: envdir /run/etc/wal-e.d/env bash /scripts/wale_restore.sh exited with code=1
2021-04-29 16:46:04,741 INFO: Lock owner: app-db-postgres-0; I am app-db-postgres-1
2021-04-29 16:46:04,757 INFO: bootstrap from leader 'app-db-postgres-0' in progress
2021-04-29 16:46:14,741 INFO: Lock owner: app-db-postgres-0; I am app-db-postgres-1
Thanks in Advance
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported bootstrap paths, /scripts/basebackup.sh and /scripts/wale_restore.sh, and review the logs showing pg_basebackup failure and the 102400MB WAL threshold. Done would require identifying why replica restoration fails for this workload and documenting or implementing a graceful resolution, but the issue does not define a specific requested change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, postgresql
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100