zalando / zalando/postgres-operator

Multiple labels in node_readiness_label & empty postgresql spec nodeAffinity triggers a perma-diff in cluster statefulset

Open
#2,931 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

  • Which image of the operator are you using? ghcr.io/zalando/postgres-operator:v1.12.2
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? Cloud, K8s, GCP & AWS
  • Are you running Postgres Operator in production? yes
  • Type of issue? Bug report

We encountered an issue where when we defined two labels in the node_readiness_label the postgresql cluster's with empty nodeAffinity constantly showed a drift in the postgresql statefulset and triggered a recreation of the STS and switchover of the database.

postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"info","msg":"reason: new statefulset's pod affinity does not match the current one","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"replacing statefulset","pkg":"cluster","time":"2025-07-01T22:34:45Z"}

here's the full debug log:

postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"info","msg":"statefulset platform-harbor/ccs-harbor-postgres is not in the desired state and needs to be updated","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-          terminationMessagePath: /dev/termination-log,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-          terminationMessagePolicy: File,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-          terminationMessagePath: /dev/termination-log,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-          terminationMessagePolicy: File,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-      restartPolicy: Always,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-      dnsPolicy: ClusterFirst,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-      serviceAccount: postgres-pod,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-                    key: postgres_ready,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"+                    key: nodepool,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-                      true","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"+                      platform","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-                    key: nodepool,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"+                    key: postgres_ready,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-                      platform","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"+                      true","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-      schedulerName: default-scheduler,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-      kind: PersistentVolumeClaim,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-      apiVersion: v1,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-      status: {","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-        phase: Pending","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-      }","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"+      status: {}","pkg":"cluster","time":"2025-07-01T22:34:45Z"}
postgres-operator-5cc587dcd5-bxdct postgres-operator {"cluster-name":"platform-harbor/ccs-harbor-postgres","level":"debug","msg":"-  revisionHistoryLimit: 10,","pkg":"cluster","time":"2025-07-01T22:34:45Z"}

The above was reproduced on multiple clusters & postgres instances with the following setup

OperatorConfiguration:

configuration:
  kubernetes:
    node_readiness_label:
      nodepool: "platform"
      postgres_ready: "true"

the cluster spec is very basic without spec.nodeAffinity

the produced StatefulSet appeared to be correct:

      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nodepool
                operator: In
                values:
                - platform
              - key: postgres_ready
                operator: In
                values:
                - "true"

however per the log above, the operator constantly detects a diff and triggers a recreation of the postgres cluster STS, from what I can tell it seem to want to flip the order of the keys in the match expression:

remove key: postgres_ready in first position
add key: nodepool in first position
remove key: nodepool in second position
add key: postgres_ready in second position

which is the order of how the keys are defined in node_readiness_label? but I might be misreading the debug log on this.

we currently worked around this issue by removing the nodepool=platform label from node_readiness_label and defined it explicitly in the postgresql spec nodeAffinity field, this solved the issue for us.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the operator's pkg/cluster logic using the reported OperatorConfiguration with two node_readiness_label entries and a cluster spec without nodeAffinity. Reproduce the StatefulSet comparison and verify that repeated reconciliations no longer report an affinity difference or recreate the StatefulSet.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.