zalando / zalando/postgres-operator

PostgreSQL Replica Fails to Start After Major Version Upgrade (WAL-G Issue)

Open
#2,957 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

The operator was upgraded from version 1.9 to 1.11, and PostgreSQL was upgraded from 13 to 14 using the python3 /scripts/inplace_upgrade.py N in-place upgrade command.

During this process, the leader node was successfully upgraded, but the replica failed to start due to a WAL-G timeline/history issue. I tried several approaches, such as deleting the replica pod, deleting PVC, and even reinitialising the replica from the leader pod, but none of these solutions worked.

The only workaround that resolved the issue was to delete the entire backup from Azure Blob Storage, create a fresh backup from the leader, and then restart the replica.

At first, I assumed this was a random error. However, when I attempted a similar upgrade on another PostgreSQL cluster, I encountered the same problem—the replica consistently failed to start after the major version upgrade.

Unfortunately, in my current environment, deleting the entire backup (as I did in the development cluster) is not an option.

Could you please suggest if there is an alternative solution to this issue?
ghcr.io/zalando/spilo-16:3.2-p2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported python3 /scripts/inplace_upgrade.py N upgrade path and the ghcr.io/zalando/spilo-16:3.2-p2 image, then trace the WAL-G timeline/history handling for replicas after PostgreSQL 13-to-14 upgrades. Compare the failed replica initialization with the fresh-backup workaround; done should identify and validate an alternative that preserves the existing Azure Blob Storage backup.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kubernetes, postgresql, python
Domain
cloud, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.