zalando / zalando/postgres-operator
Operator cache obsolete IP address during cluster migration
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
Please, answer some short questions which should help us to understand your problem / question better?
- Which image of the operator are you using? e.g. ghcr.io/zalando/postgres-operator:v1.15.1
- Where do you run it - cloud or metal? Kubernetes or OpenShift? DigitalOcean managed K8S
- Are you running Postgres Operator in production? yes
- Type of issue? Bug report
We observed operator malfunction during k8s nodes rotation - it recreates database pod and tried to connect to its old IP address:
Before restart:
asana-automate-db-0 2/2 Running 0 19d 10.244.3.71 psql-staging-b248b8-p7zrb <none> <none>
After restart:
asana-automate-db-0 2/2 Running 0 16m 10.244.15.254 psql-staging-b248b8-dxqfq <none> <none>
Operator logs:
time="2026-03-23T16:24:16Z" level=warning msg="migrating single pod cluster \"sprint-reports/asana-automate-db\", this will cause downtime of the Postgres cluster until pod is back" cluster-name=sprint-reports/asana-automate-db pkg=cluster
time="2026-03-23T16:24:16Z" level=info msg="moving pod \"sprint-reports/asana-automate-db-0\" out of the end-of-life node \"psql-staging-b248b8-p7zrb\"" cluster-name=sprint-reports/asana-automate-db pkg=cluster
time="2026-03-23T16:24:16Z" level=debug msg="subscribing to pod \"sprint-reports/asana-automate-db-0\"" cluster-name=sprint-reports/asana-automate-db pkg=cluster
time="2026-03-23T16:25:00Z" level=info msg="pod \"sprint-reports/asana-automate-db-0\" has been recreated" cluster-name=sprint-reports/asana-automate-db pkg=cluster
time="2026-03-23T16:25:00Z" level=debug msg="unsubscribing from pod \"sprint-reports/asana-automate-db-0\" events" cluster-name=sprint-reports/asana-automate-db pkg=cluster
time="2026-03-23T16:25:00Z" level=info msg="pod \"sprint-reports/asana-automate-db-0\" moved from node \"psql-staging-b248b8-p7zrb\" to node \"psql-staging-b248b8-dxqfq\"" cluster-name=sprint-reports/asana-automate-db pkg=cluster
time="2026-03-23T16:25:00Z" level=debug msg="subscribing to pod \"sprint-reports/asana-automate-db-0\"" cluster-name=sprint-reports/asana-automate-db pkg=cluster
time="2026-03-23T16:25:00Z" level=debug msg="switching over from \"asana-automate-db-0\" to \"sprint-reports/asana-automate-db-0\"" cluster-name=sprint-reports/asana-automate-db pkg=cluster
time="2026-03-23T16:25:00Z" level=debug msg="making POST http request: http://10.244.3.71:8008/switchover" cluster-name=sprint-reports/asana-automate-db pkg=cluster
time="2026-03-23T16:25:30Z" level=debug msg="unsubscribing from pod \"sprint-reports/asana-automate-db-0\" events" cluster-name=sprint-reports/asana-automate-db pkg=cluster
time="2026-03-23T16:25:30Z" level=error msg="could not switchover to pod \"sprint-reports/asana-automate-db-0\": could not switch over from \"asana-automate-db-0\" to \"sprint-reports/asana-automate-db-0\": could not make request: Post \"http://10.244.3.71:8008/switchover\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" cluster-name=sprint-reports/asana-automate-db pkg=cluster
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the cluster migration flow shown in the operator logs, especially pod recreation, event subscription, and the switchover request. Inspect how the pod IP is retained or refreshed between the old and new pods. Done means migration uses the recreated pod's current IP rather than 10.244.3.71 and the reported switchover failure is prevented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, postgresql
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100