zalando / zalando/postgres-operator

Clusters being restarted at every sync interval

Open
#2,909 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? ghcr.io/zalando/postgres-operator:v1.14.0
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? OCI K8S
  • Are you running Postgres Operator in production? yes
  • Type of issue? question

It looks like the operator do a cluster syncing at every 30 minutes, but each time this happens, my clusters begins to restart, resulting in some downtimes on my production environment. I don't know if it's a bug or a misconfiguration from my manifests.

Here's my PostgreSQL production environment configuration:

apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
  name: collection-pg-cluster-primary
  namespace: pg-primary
spec:
  spiloFSGroup: 103
  teamId: "collection-pg-cluster-id"
  volume:
    size: 50Gi
  numberOfInstances: 3
  env:
    - name: USE_WALG_BACKUP
      value: "true"
    - name: WAL_S3_BUCKET
      value: collection-pg-backups
    - name: AWS_REGION
      value: nyc3
    - name: AWS_ENDPOINT
      value: https://nyc3.digitaloceanspaces.com
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: pod-secret
          key: AWS_ACCESS_KEY_ID
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: pod-secret
          key: AWS_SECRET_ACCESS_KEY
  users:
    collection:
      - superuser
      - createdb
    grafana:
      - login
      - replication
    replicator:
      - replication
  databases:
    collection: collection
  enableLogicalBackup: false
  enableMasterLoadBalancer: true
  postgresql:
    version: "14"
    parameters:
      password_encryption: scram-sha-256
      log_statement: "all"
      wal_level: logical
      max_replication_slots: "80"
      max_wal_senders: "80"
  patroni:
    slots:
      collection_sub:
        database: collection
        plugin: pgoutput
        type: logical
  tls:
    secretName: pg-primary-tls
    caFile: "ca.crt"
  resources:
    requests:
      cpu: "2"
      memory: 4Gi
    limits:
      cpu: "4"
      memory: 8Gi
  sidecars:
    - name: "exporter"
      image: "quay.io/prometheuscommunity/postgres-exporter:latest"
      ports:
        - name: metrics
          containerPort: 9187
          protocol: TCP
      resources:
        limits:
          cpu: 500m
          memory: 256M
        requests:
          cpu: 100m
          memory: 200M
      env:
        - name: "DATA_SOURCE_URI"
          value: "127.0.0.1:5432?sslmode=disable"
        - name: "DATA_SOURCE_USER"
          value: "$(POSTGRES_USER)"
        - name: "DATA_SOURCE_PASS"
          value: "$(POSTGRES_PASSWORD)"
    - name: "logs-collector"
      image: "sa-saopaulo-1.ocir.io/grqi4cogacpe/collection-images/postgres-log-exporter:v0.3.8"
      resources:
        limits:
          memory: 1Gi
        requests:
          cpu: 500m
          memory: 512Mi

And the operator's logs:

time="2025-05-15T11:48:40Z" level=debug msg="syncing connection pooler (master, replica) from (false, nil) to (false, nil)" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:40Z" level=debug msg="closing database connection" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:40Z" level=debug msg="closing database connection" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:40Z" level=debug msg="syncing roles" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:40Z" level=debug msg="syncing pod disruption budgets" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:40Z" level=debug msg="making GET http request: http://10.0.11.252:8008/patroni" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:40Z" level=debug msg="making GET http request: http://10.0.11.93:8008/patroni" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:40Z" level=debug msg="making GET http request: http://10.0.10.71:8008/patroni" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:40Z" level=debug msg="making GET http request: http://10.0.11.93:8008/config" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:40Z" level=debug msg="making GET http request: http://10.0.10.71:8008/config" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:39Z" level=debug msg="syncing Patroni config" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:39Z" level=debug msg="closing database connection" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:39Z" level=debug msg="syncing statefulsets" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:39Z" level=info msg="Mount additional volumes: [{Name:pg-primary-tls MountPath:/tls SubPath: IsSubPathExpr:<nil> TargetContainers:[postgres] VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:&SecretVolumeSource{SecretName:pg-primary-tls,Items:[]KeyToPath{},DefaultMode:*416,Optional:nil,} NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:39Z" level=debug msg="closing database connection" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:39Z" level=debug msg="syncing roles" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:39Z" level=debug msg="syncing pod disruption budgets" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:39Z" level=debug msg="making GET http request: http://10.0.10.83:8008/patroni" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:39Z" level=debug msg="syncing Patroni config" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:39Z" level=info msg="cluster version up to date. current: 140015, min desired: 140000" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:39Z" level=debug msg="closing database connection" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:39Z" level=debug msg="closing database connection" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="syncing roles" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="volume claims have been synced successfully" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:38Z" level=info msg="cluster version up to date. current: 140015, min desired: 140000" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="closing database connection" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="closing database connection" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="syncing statefulsets" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:38Z" level=info msg="Mount additional volumes: [{Name:pg-primary-read-replica-tls MountPath:/tls SubPath: IsSubPathExpr:<nil> TargetContainers:[postgres] VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:&SecretVolumeSource{SecretName:pg-primary-read-replica-tls,Items:[]KeyToPath{},DefaultMode:*416,Optional:nil,} NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="syncing roles" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="syncing pod disruption budgets" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="making GET http request: http://10.0.11.147:8008/patroni" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="syncing Patroni config" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="volume claim for volume \"pgdata-collection-pg-cluster-primary-2\" do not require updates" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="syncing pod disruption budgets" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="making GET http request: http://10.0.11.162:8008/patroni" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="syncing Patroni config" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:38Z" level=debug msg="volume claim for volume \"pgdata-collection-pg-cluster-primary-1\" do not require updates" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:37Z" level=debug msg="syncing statefulsets" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:37Z" level=info msg="Mount additional volumes: [{Name:pg-secondary-tls MountPath:/tls SubPath: IsSubPathExpr:<nil> TargetContainers:[postgres] VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:&SecretVolumeSource{SecretName:pg-secondary-tls,Items:[]KeyToPath{},DefaultMode:*416,Optional:nil,} NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:37Z" level=debug msg="volume claim for volume \"pgdata-collection-pg-cluster-primary-0\" do not require updates" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:37Z" level=debug msg="syncing statefulsets" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:37Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:37Z" level=debug msg="volume claims have been synced successfully" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:36Z" level=debug msg="syncing collection-pg-cluster-primary-failover endpoint" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:36Z" level=debug msg="volume claim for volume \"pgdata-read-replica-0\" do not require updates" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:36Z" level=debug msg="syncing collection-pg-cluster-primary-sync endpoint" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:36Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:36Z" level=debug msg="volume claims have been synced successfully" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:35Z" level=debug msg="syncing collection-pg-cluster-primary-config endpoint" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:35Z" level=debug msg="syncing read-replica-failover endpoint" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:35Z" level=debug msg="volume claim for volume \"pgdata-collection-pg-cluster-secondary-0\" do not require updates" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:35Z" level=debug msg="volume claims have been synced successfully" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:35Z" level=debug msg="syncing collection-pg-cluster-primary-leader endpoint" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:35Z" level=debug msg="syncing read-replica-sync endpoint" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:35Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:35Z" level=debug msg="volume claim for volume \"pgdata-metabase-pg-0\" do not require updates" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:35Z" level=debug msg="syncing collection-pg-cluster-primary-config service" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:35Z" level=debug msg="syncing read-replica-config endpoint" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:34Z" level=debug msg="syncing collection-pg-cluster-secondary-failover endpoint" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:34Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:34Z" level=debug msg="syncing read-replica-leader endpoint" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:34Z" level=debug msg="syncing collection-pg-cluster-secondary-sync endpoint" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:34Z" level=debug msg="syncing metabase-pg-failover endpoint" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:34Z" level=debug msg="syncing read-replica-config service" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:34Z" level=debug msg="syncing collection-pg-cluster-secondary-config endpoint" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:34Z" level=debug msg="syncing metabase-pg-sync endpoint" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:33Z" level=debug msg="syncing replica service" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:33Z" level=debug msg="final load balancer source ranges as seen in a service spec (not necessarily applied): [\"127.0.0.1/32\"]" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:33Z" level=debug msg="syncing collection-pg-cluster-secondary-leader endpoint" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:33Z" level=debug msg="syncing metabase-pg-config endpoint" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:33Z" level=debug msg="syncing collection-pg-cluster-secondary-config service" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:33Z" level=debug msg="syncing metabase-pg-leader endpoint" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:33Z" level=debug msg="syncing replica service" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:32Z" level=debug msg="syncing metabase-pg-config service" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:32Z" level=debug msg="syncing master service" cluster-name=pg-primary/collection-pg-cluster-primary pkg=cluster
time="2025-05-15T11:48:32Z" level=debug msg="syncing replica service" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:31Z" level=debug msg="syncing master service" cluster-name=pg-primary-read/read-replica pkg=cluster
time="2025-05-15T11:48:31Z" level=debug msg="syncing replica service" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:30Z" level=debug msg="syncing master service" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=cluster
time="2025-05-15T11:48:30Z" level=debug msg="syncing master service" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:29Z" level=info msg="syncing of the cluster started" cluster-name=pg-secondary/collection-pg-cluster-secondary pkg=controller worker=3
time="2025-05-15T11:48:29Z" level=debug msg="team API is disabled" cluster-name=metabase/metabase-pg pkg=cluster
time="2025-05-15T11:48:29Z" level=info msg="syncing of the cluster started" cluster-name=metabase/metabase-pg pkg=controller worker=0
time="2025-05-15T11:48:29Z" level=info msg="SYNC event has been queued" cluster-name=pg-primary-read/read-replica pkg=controller worker=1
time="2025-05-15T11:48:29Z" level=info msg="SYNC event has been queued" cluster-name=metabase/metabase-pg pkg=controller worker=0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by correlating the reported 30-minute sync with Kubernetes pod restart events and the StatefulSet and Patroni activity in the supplied operator logs. Compare the PostgreSQL manifest with the operator's reconciliation output, including the additional TLS volumes. Done means identifying whether synchronization changes the pods and documenting or fixing the cause of the restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.