zalando / zalando/postgres-operator

Not restoring from backup because of amount of generated wals

Open
#1,477 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.6.0
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? [Bare Metal K8s]
  • Are you running Postgres Operator in production? yes
  • Type of issue? question

Please give me an advice. I get error during bootstrapping the replica from leader after I reinit the patroni in replica. It can not restore from WAL because of size of last backup. How can I solve this problem gracefully? WAL backups are stored in S3 storage. Here is my logs from replica:

2021-04-29 16:45:44,741 INFO: Lock owner: app-db-postgres-0; I am app-db-postgres-1
2021-04-29 16:45:44,741 INFO: bootstrap from leader 'app-db-postgres-0' in progress
pg_basebackup: error: COPY stream ended before last file was finished
pg_basebackup: removing data directory "/home/postgres/pgdata/pgroot/data"
2021-04-29 16:45:54,113 ERROR: Error creating replica using method basebackup_fast_xlog: /scripts/basebackup.sh exited with code=1
2021-04-29 16:45:54,113 ERROR: failed to bootstrap from leader 'app-db-postgres-0'
2021-04-29 16:45:54,113 INFO: Removing data directory: /home/postgres/pgdata/pgroot/data
2021-04-29 16:45:54,741 INFO: Lock owner: app-db-postgres-0; I am app-db-postgres-1
2021-04-29 16:45:54,761 INFO: trying to bootstrap from leader 'app-db-postgres-0'
Current cluster size: 72570816836
Wals generated since the last backup: 305608353752
not restoring from backup because of amount of generated wals exceeds 102400MB
2021-04-29 16:45:55,585 ERROR: Error creating replica using method wal_e: envdir /run/etc/wal-e.d/env bash /scripts/wale_restore.sh exited with code=1
2021-04-29 16:46:04,741 INFO: Lock owner: app-db-postgres-0; I am app-db-postgres-1
2021-04-29 16:46:04,757 INFO: bootstrap from leader 'app-db-postgres-0' in progress
2021-04-29 16:46:14,741 INFO: Lock owner: app-db-postgres-0; I am app-db-postgres-1

Thanks in Advance

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported bootstrap paths, /scripts/basebackup.sh and /scripts/wale_restore.sh, and review the logs showing pg_basebackup failure and the 102400MB WAL threshold. Done would require identifying why replica restoration fails for this workload and documenting or implementing a graceful resolution, but the issue does not define a specific requested change.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.