zalando / zalando/postgres-operator

Ignoring some fields' differences to avoid uneeded pods switchover

Open
#2,436 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? ghcr.io/zalando/spilo-15:3.0-p1
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
  • Are you running Postgres Operator in production? Not yet
  • Type of issue? Bug report, question

Every 10/30 minutes my DB pods are restarted which make the DB unavailable for a couple seconds. I think I've traced the issue
to the way the operator compares the pod in the cluster and what it expects to find.
I have a software running in my cluster that patches pods images to redirect it to a caching proxy. (say docker.io/library/toto -> mycachingproxy/docker.io/library/toto) which I think causes this.

So my question is : is my analysis correct ? I'd be glad to post a MR to fix this issue if that's the case.

Thanks by advance.

Logs from the operator :

postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:00:31Z" level=info msg="cluster has been synced" cluster-name=systems-services-harbor/harbor-db pkg=controller worker=0
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:53Z" level=info msg="SYNC event has been queued" cluster-name=systems-services-harbor/harbor-db pkg=controller worker=0
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:53Z" level=info msg="there are 1 clusters running" pkg=controller
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:53Z" level=info msg="syncing of the cluster started" cluster-name=systems-services-harbor/harbor-db pkg=controller worker=0
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:53Z" level=debug msg="team API is disabled" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:53Z" level=debug msg="team API is disabled" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:53Z" level=info msg="syncing secrets" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:53Z" level=debug msg="syncing master service" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:53Z" level=debug msg="final load balancer source ranges as seen in a service spec (not necessarily applied): [\"127.0.0.1/32\"]" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:53Z" level=debug msg="syncing replica service" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:54Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:54Z" level=info msg="volume claims do not require changes" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:54Z" level=debug msg="syncing statefulsets" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:54Z" level=debug msg="mark rolling update annotation for harbor-db-0: reason pod not yet restarted due to lazy update" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:54Z" level=debug msg="mark rolling update annotation for harbor-db-1: reason pod not yet restarted due to lazy update" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:55Z" level=debug msg="syncing Patroni config" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:55Z" level=debug msg="making GET http request: http://10.233.69.103:8008/config" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:55Z" level=debug msg="making GET http request: http://10.233.67.21:8008/config" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:55Z" level=debug msg="making GET http request: http://10.233.69.103:8008/patroni" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:55Z" level=debug msg="making GET http request: http://10.233.67.21:8008/patroni" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:55Z" level=debug msg="performing rolling update" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:55Z" level=info msg="there are 2 pods in the cluster to recreate" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:29:55Z" level=debug msg="subscribing to pod \"systems-services-harbor/harbor-db-0\"" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:11Z" level=info msg="pod \"systems-services-harbor/harbor-db-0\" has been recreated" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:11Z" level=debug msg="unsubscribing from pod \"systems-services-harbor/harbor-db-0\" events" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:11Z" level=debug msg="making GET http request: http://10.233.67.21:8008/cluster" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:11Z" level=debug msg="switching over from \"harbor-db-1\" to \"systems-services-harbor/harbor-db-0\"" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:11Z" level=debug msg="subscribing to pod \"systems-services-harbor/harbor-db-0\"" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:11Z" level=debug msg="making POST http request: http://10.233.67.21:8008/failover" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:13Z" level=debug msg="successfully switched over from \"harbor-db-1\" to \"systems-services-harbor/harbor-db-0\"" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:14Z" level=debug msg="unsubscribing from pod \"systems-services-harbor/harbor-db-0\" events" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:14Z" level=info msg="recreating old master pod \"systems-services-harbor/harbor-db-1\"" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:14Z" level=debug msg="subscribing to pod \"systems-services-harbor/harbor-db-1\"" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:30Z" level=info msg="pod \"systems-services-harbor/harbor-db-1\" has been recreated" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:30Z" level=debug msg="unsubscribing from pod \"systems-services-harbor/harbor-db-1\" events" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:30Z" level=debug msg="syncing pod disruption budgets" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:30Z" level=debug msg="syncing roles" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:30Z" level=debug msg="closing database connection" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:30Z" level=debug msg="syncing databases" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:30Z" level=debug msg="closing database connection" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:30Z" level=debug msg="syncing prepared databases with schemas" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:30Z" level=debug msg="syncing connection pooler (master, replica) from (true, nil) to (true, nil)" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:30Z" level=debug msg="final load balancer source ranges as seen in a service spec (not necessarily applied): [\"127.0.0.1/32\"]" cluster-name=systems-services-harbor/harbor-db pkg=cluster
postgres-operator-56fb8b4bdb-zp7nw postgres-operator time="2023-10-03T12:30:30Z" level=info msg="cluster has been synced" cluster-name=systems-services-harbor/harbor-db pkg=controller worker=0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the operator's “syncing statefulsets” and “performing rolling update” paths described in the logs, focusing on how pod differences trigger recreation. Confirm whether the caching-proxy image mutation is treated as drift, then add coverage showing that this mutation does not cause unnecessary pod recreation or switchover.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.