zalando / zalando/postgres-operator

pooler deployment is not recreated

Open
#2,446 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.10.1
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? AWS and Azure
  • Are you running Postgres Operator in production? yes
  • Type of issue? Bug report

Postgres Operator does not recreate pooler deployment if it existed in the kubernetes cluster before. Steps to reproduce:

  1. Create Postgres operator deployment
  2. Create a postgresqlCluster with the following parameters:
      numberOfInstances: 3
      enableConnectionPooler: true
      enableMasterLoadBalancer: false
      enableMasterPoolerLoadBalancer: false
      postgresql:
        version: "15"
  1. Remove (disable) Postgres operator deployment.
  2. recreate deployment and cluster from steps 1 and 2
    At this point postgres-ha-pooler Deployment and Kubernetes service are missing. Postgres pods fail to start without pooler. Postgres operator related logs are:

time="2023-10-16T13:55:12Z" level=info msg="   \"Schema\": \"pooler\"," pkg=controller
time="2023-10-16T13:55:12Z" level=info msg="   \"User\": \"pooler\"," pkg=controller
... 
time="2023-10-16T13:55:12Z" level=info msg="   \"ConnectionPoolerDefaultCPURequest\": \"500m\"," pkg=controller
time="2023-10-16T13:55:12Z" level=info msg="   \"ConnectionPoolerDefaultMemoryRequest\": \"100Mi\"," pkg=controller
time="2023-10-16T13:55:12Z" level=info msg="   \"ConnectionPoolerDefaultCPULimit\": \"1000m\"," pkg=controller
time="2023-10-16T13:55:12Z" level=info msg="   \"ConnectionPoolerDefaultMemoryLimit\": \"512Mi\"," pkg=controller
...
time="2023-10-16T13:55:12Z" level=info msg="   \"EnableMasterPoolerLoadBalancer\": false," pkg=controller
time="2023-10-16T13:55:12Z" level=info msg="   \"EnableReplicaPoolerLoadBalancer\": false," pkg=controller
...
time="2023-10-16T13:55:13Z" level=warning msg="could not connect to Postgres database: dial tcp 172.20.205.79:5432: connect: connection refused"

Expected behavior: Postgres operator recreates pooler deployment and service properly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the restart scenario using the listed cluster parameters and operator image, then trace the reconciliation of the pooler Deployment and Kubernetes service while checking the operator logs for the database connection failure. Done means the operator recreates both resources after the operator deployment and cluster are recreated, allowing the Postgres pods to start.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.