zalando / zalando/postgres-operator
inplace upgrade skipping major version upgrade, replica pod is not streaming from primary
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
- Which image of the operator are you using? ghcr.io/zalando/postgres-operator:v1.15.1
- **Where do you run it - cloud or metal? ** AWS EKS
- Are you running Postgres Operator in production? yes
- Type of issue? Bug report
I upgraded my PostgreSQL cluster with the following changes:
- Replaced the Spilo image.
- Updated some ENV values.
- Updated PostgreSQL version from 16 to 17.
The operator did everything well until it checked the state of the patron node before upgrading. It retrieved the patroni node's status around 1 second before the node entered state streaming.
So it failed https://github.com/zalando/postgres-operator/blob/v1.15.1/pkg/cluster/majorversionupgrade.go#L227
Is there now a mechanism for operators to wait for the Patroni cluster to be ready before upgrading, or to issue a de resync event immediately?
Noted: temporary solution is restated the operator.
Operator logs:
`time="2026-06-05T09:25:51Z" level=info msg="reason: new statefulset containers's postgres (index 0) environment does not match the current one" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:25:51Z" level=info msg="reason: new statefulset's pod template metadata annotations
time="2026-06-05T09:25:51Z" level=debug msg="replacing statefulset" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:25:51Z" level=debug msg="waiting for the statefulset to be deleted" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:26:51Z" level=debug msg="syncing Patroni config" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:26:51Z" level=debug msg="making GET http request: http://10.100.0.179:8008/config" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:26:51Z" level=debug msg="making GET http request: http://10.100.6.183:8008/config" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:26:51Z" level=debug msg="making GET http request: http://10.100.46.176:8008/config" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:26:51Z" level=debug msg="making GET http request: http://10.100.0.179:8008/patroni" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:26:51Z" level=debug msg="making GET http request: http://10.100.46.176:8008/patroni" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:26:51Z" level=debug msg="making GET http request: http://10.100.6.183:8008/patroni" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:26:51Z" level=info msg="performing rolling update" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:26:51Z" level=info msg="there are 3 pods in the cluster to recreate" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:26:51Z" level=debug msg="subscribing to pod \"okela-datastore/okela-postgresql-0\"" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:16Z" level=info msg="pod \"okela-datastore/okela-postgresql-0\" has been recreated" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:16Z" level=debug msg="unsubscribing from pod \"okela-datastore/okela-postgresql-0\" events" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:16Z" level=debug msg="subscribing to pod \"okela-datastore/okela-postgresql-2\"" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:41Z" level=info msg="pod \"okela-datastore/okela-postgresql-2\" has been recreated" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:41Z" level=debug msg="unsubscribing from pod \"okela-datastore/okela-postgresql-2\" events" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:41Z" level=debug msg="making GET http request: http://10.100.6.183:8008/cluster" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:41Z" level=debug msg="subscribing to pod \"okela-datastore/okela-postgresql-0\"" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:41Z" level=debug msg="switching over from \"okela-postgresql-1\" to \"okela-datastore/okela-postgresql-0\"" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:41Z" level=debug msg="making POST http request: http://10.100.6.183:8008/switchover" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:43Z" level=debug msg="successfully switched over from \"okela-postgresql-1\" to \"okela-datastore/okela-postgresql-0\"" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:45Z" level=debug msg="unsubscribing from pod \"okela-datastore/okela-postgresql-0\" events" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:45Z" level=info msg="recreating old master pod \"okela-datastore/okela-postgresql-1\"" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:27:45Z" level=debug msg="subscribing to pod \"okela-datastore/okela-postgresql-1\"" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=info msg="pod \"okela-datastore/okela-postgresql-1\" has been recreated" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=debug msg="unsubscribing from pod \"okela-datastore/okela-postgresql-1\" events" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=debug msg="syncing roles" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=debug msg="closing database connection" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=debug msg="syncing connection pooler (master, replica) from (false, nil) to (false, nil)" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=debug msg="making GET http request: http://10.100.0.174:8008/patroni" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=debug msg="making GET http request: http://10.100.6.162:8008/patroni" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=debug msg="making GET http request: http://10.100.46.161:8008/patroni" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=debug msg="making GET http request: http://10.100.0.174:8008/cluster" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=debug msg="making GET http request: http://10.100.0.174:8008/patroni" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=info msg="skipping major version upgrade, replica okela-postgresql-1 is not streaming from primary" cluster-name=okela-datastore/okela-postgresql pkg=cluster
time="2026-06-05T09:28:18Z" level=info msg="cluster has been updated" cluster-name=okela-datastore/okela-postgresql pkg=controller worker=0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pkg/cluster/majorversionupgrade.go around line 227 and trace the Patroni status checks used during the inplace major-version upgrade. Use the supplied operator logs and Patroni cluster state as the reproduction context; done means the upgrade handles a replica that becomes streaming shortly after the state check without incorrectly skipping the major-version upgrade.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes, postgresql
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100