Feature: Allow custom podAffinity and podAntiAffinity rules on postgres pods

Feature request

This feature request involves enabling advanced pod scheduling capabilities through the following

• Enabling custom podAffinity rules on postgres pods
• Enabling custom podAntiAffinity rules on postgres pods

Problem description

Currently the postgresql crd only allows specifying the anti affinity topology constraint for scheduling the postgres pods to specific nodes. Due to this limitation it is not possible to further configure affinity and anti affinity rules for these pods.

These are important features for many use cases where pods should be scheduled according to further constraints. An example of such a case would be to select nodes according to selected resource allocation strategies defined by labels, such as using a dedicated node for a postrges instance (without resource allocation constraints) vs a node where postgres instances can share resources with other pods according to resource requests and limits.

A postgres instance schedulable to a dedicated instance:

spec:
  # Allows the pod to be scheduled on a node with specified taint 
  # The taint being 'cluster-resource-strategy=dedicated:NoSchedule'
  tolerations:
    - key: cluster-resource-strategy
      operator: Equal
      value: dedicated
      effect: NoSchedule
  affinity:
   # Ensures that the pod is scheduled on a node with the same resource strategy
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: cluster-resource-strategy
                operator: In
                values:
                  - dedicated
    # Prevents other pods with the same 'dedicated' label from being scheduled
    # effectively making it a "dedicated" node
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              cluster-resource-strategy: dedicated
          topologyKey: kubernetes.io/hostname # Default in current postgresql deployment

A postgres instance schedulable to shared nodes:

affinity:
  # Ensures that the pod is scheduled on a node with the same resource strategy
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: cluster-resource-strategy
              operator: In
              values:
                - shared

Additional information

• Which image of the operator are you using?

  ghcr.io/zalando/spilo-16:3.3-p1

• Where do you run it - cloud or metal? Kubernetes or OpenShift?

  anywhere

• Are you running Postgres Operator in production?

  yes

• Type of issue?

  Feature request

Start with the PostgreSQL CRD and the operator path that creates or updates PostgreSQL pods. Trace how the existing anti-affinity topology constraint is represented and applied, then determine how custom podAffinity and podAntiAffinity should be exposed. Done means valid CRD configuration is accepted and both rule types are applied to the resulting pods, with coverage in the relevant operator tests.