Split Affinity for Pooler and Postgres Pods

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using?: registry.opensource.zalan.do/acid/postgres-operator:v1.10.1
• Where do you run it - cloud or metal? Kubernetes or OpenShift?: AWS EKS, Azure AKS
• Are you running Postgres Operator in production?: yes
• Type of issue?: feature request

As described in #1600 pooler deployment gets stuck in roll-out with requiredDuringSchedulingIgnoredDuringExecution affinity.
The implemented and proposed solution is to use preferredDuringSchedulingIgnoredDuringExecution affinity both for pooler and postgres pods. However, this creates an unnecessary coupling between two different deployments which may have different SLAs, zone-redundancy requirements and roll-out strategies.

Since pooler is a Deployment and postgres is a StatefulSet the expectation is that postgres StatefulSet followsrequiredDuringSchedulingIgnoredDuringExecution to guarantee that Pods and related PVC are distributed across 3 availability zones. However, this affinity does not work for pooler Deployment for which preferredDuringSchedulingIgnoredDuringExecution affinity must be used to schedule pods in a cluster with Availability Zones.

Solution: decouple the affinity setting for Pooler from Postgres StatefulSet.

Start by tracing how affinity is configured for the pooler Deployment and Postgres StatefulSet, using issue #1600 for the scheduling context. The work is done when each deployment can use its own affinity policy, preserving required affinity for Postgres and preferred affinity for the pooler, with the relevant operator checks passing.