Race-condition when doing syncRoles()

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? master branch
• Where do you run it - cloud or metal? Kubernetes or OpenShift? Cloud (on-prem K8s) and kind (local)
• Are you running Postgres Operator in production? yes
• Type of issue? Bug report

Description

When a Postgres cluster undergoes an update that requires a rolling update of pods, the operator hits a race condition during the role synchronization phase.

It attempts to execute syncRoles(), but because Postgres is still starting up and temporarily in recovery mode, the database is read-only. This causes the sync to fail.

https://github.com/zalando/postgres-operator/blob/4d40270890a46aac133f45339576b39b797a8ae9/pkg/cluster/cluster.go#L1147-L1151

Evidence & Logs

The operator logs the following during the rollout with an 25006 (read-only transaction) error:

postgres-operator-758c4b864f-6slj6 postgres-operator time="2026-05-27T19:38:33Z" level=debug msg="syncing roles" cluster-name=default/cafebabe pkg=cluster worker=0
postgres-operator-758c4b864f-6slj6 postgres-operator time="2026-05-27T19:38:33Z" level=debug msg="closing database connection" cluster-name=default/cafebabe pkg=cluster worker=0
postgres-operator-758c4b864f-6slj6 postgres-operator time="2026-05-27T19:38:33Z" level=error msg="could not sync roles: error executing sync statements: could not execute sync requests for users: could not alter user \"postgres\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not alter user \"standby\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not alter user \"monitoring_owner\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not set custom user \"monitoring_owner\" parameters: pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not alter user \"monitoring_reader\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not set custom user \"monitoring_reader\" parameters: pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not alter user \"monitoring_writer\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not set custom user \"monitoring_writer\" parameters: pq: cannot execute ALTER ROLE in a read-only transaction (25006)', 'could not alter user \"monitoring\": pq: cannot execute ALTER ROLE in a read-only transaction (25006)" cluster-name=default/cafebabe pkg=cluster worker=0

Impact

This race condition sets the cluster status to PostgresClusterStatus: UpdateFailed, even though the Kubernetes events confirm the rollout was successful:
Normal Update 2m postgres-operator Rolling update done - pods have been recreated

In the code for the sync of the roles, there is no check for the status code or a retry logic to mark this as transient issue instead of a UpdateFailed error.

https://github.com/zalando/postgres-operator/blob/4d40270890a46aac133f45339576b39b797a8ae9/pkg/cluster/sync.go#L1508-L1511

During the next reconciliation cycle, the database finishes recovery, syncRoles() succeeds, and the status returns to Running. However, this transient failure causes flapping statuses and false-positive alerts.

Suggested Fix

Since role syncing is non-critical during a brief startup window (especially if roles haven't changed), the operator should check if the database is in recovery mode or catch the 25006 error code and handle it gracefully rather than failing the entire update status.

I am happy to open a Pull Request to add a check for the recovery state if you'd like!

Start in pkg/cluster/cluster.go at the linked syncRoles() call, then read pkg/cluster/sync.go at the linked role-sync error handling. Trace how PostgreSQL recovery and error 25006 become UpdateFailed; done means this transient read-only window no longer causes a failed update while later role synchronization still succeeds.