spilo-role replica label missing from one of the patroni replica pods

• Operator Version? - ghcr.io/zalando/postgres-operator:v1.8.2
• Where do you run it - cloud or metal? Bare-metal K8s
• Type of issue? Bug Report

Hi,

I saw that one of the patroni replica pods was stuck in "starting" state because it was not able to connect via the replication k8s service. I checked that was because the endpoints were missing for this k8s service.

The endpoint wasn't created because none of the patroni pods had spilo-role=replica label set, so the endpoint isn't able to select any target

postgres operator logs

time="2024-07-05T09:36:00Z" level=debug msg="making GET http request: http://[fd74:ca9b:3a09:868c:172:18:0:6844]:8008/patroni" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="making GET http request: http://[fd74:ca9b:3a09:868c:172:18:0:6643]:8008/patroni" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="syncing pod disruption budgets" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="syncing roles" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="closing database connection" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="syncing databases" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="closing database connection" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="syncing prepared databases with schemas" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="syncing connection pooler (master, replica) from (false, nil) to (false, nil)" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=info msg="cluster has been synced" cluster-name= pkg=controller worker=0

whenever I try to label the pod manually, postgres-operator removes the spilo-role label

I don't see anything useful in the logs

The issue names no files or tests. Start by tracing how the operator reconciles Patroni replica pod labels and how the replication Kubernetes Service selects endpoints, using the reported v1.8.2 behavior as the reproduction case. Done means replica pods retain the expected spilo-role=replica label and the replication Service gets endpoints without manual labeling.