EventRepair never gets enqueued for UpdateFailed

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? postgres-operator:v1.8.0
• Where do you run it - cloud or metal? Kubernetes or OpenShift? VMware Tanzu k8s
• Are you running Postgres Operator in production? Yes
• Type of issue? Bug report/Question

Issue:

I followed the given instructions to promote the standby cluster i.e. patronictl edit-config to remove the standby section and then triggering pod restart by removing the standby field in the postgresql CR. The CR goes into Updating state but then as the pod comes back up, the containers are 4/5 ready and running and zalando operator starts the sync process and throws below error plus marks the cluster state as UpdateFailed.

time="2023-05-10T05:32:34Z" level=error msg="could not sync roles: error executing sync statements: could not execute sync requests for users: could not create user \"superuser\": pq: cannot execute CREATE ROLE in a read-only transaction" cluster-name=default/swat-standby-redb-20 pkg=cluster worker=0

Actual:
It waits for resync_period i.e. default 30 mins to sync again and then updates the cluster state to Running.

Expected:
It should wait for default repair_period i.e. 5 mins and sync again so that it brings the cluster in Running state sooner.

Notes:
This is similar to https://github.com/zalando/postgres-operator/issues/2102 and I have added a comment there too. Reiterating my point here:

• Why does it wait for sync_period?
• Shouldn't a repair be invoked in case of UpdateFailed status?
• Wasn't repair feature introduced in order to tackle syncing of failed clusters rather than syncing all clusters in bulk?

I checked the zalando postgres operator code and I expected that the EventRepair event type will get queued once the update fails since the function NeedsRepair includes the ClusterStatus ClusterStatusUpdateFailed as one of the repair success conditions. However, it doesn't. Let me know if I am missing something here.

References:
NeedsRepair func: https://github.com/zalando/postgres-operator/blob/af084a5a650527c43f0c0fc579551a741e77f5c8/pkg/cluster/cluster.go#L1050

Success func: https://github.com/zalando/postgres-operator/blob/af084a5a650527c43f0c0fc579551a741e77f5c8/pkg/apis/acid.zalan.do/v1/util.go#L90

Start with NeedsRepair in pkg/cluster/cluster.go and Success in pkg/apis/acid.zalan.do/v1/util.go, then trace how UpdateFailed events are queued and how repair_period versus resync_period is selected. Done means an UpdateFailed cluster receives EventRepair and retries on the repair interval, with the behavior covered by the relevant existing tests if available.