percona / percona/percona-postgresql-operator
PerconaPGCluster standby removal not propagated to inner PostgresCluster — cluster stuck in standby after spec.standby removed
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 385
- Forks
- 83
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 39
Description
Report
When a PerconaPGCluster is configured with spec.standby and then the spec.standby block is removed (to promote the cluster to primary), the Percona operator reconciles the outer CR but does not update the inner upstream.pgv2.percona.com/v1beta1 PostgresCluster object it manages. The inner object retains spec.standby.enabled: true, which keeps the Patroni DCS standby_cluster config in place and leaves PostgreSQL in recovery mode indefinitely.
More about the problem
Expected behavior
Removing spec.standby from PerconaPGCluster should cause the operator to update the inner PostgresCluster to remove spec.standby, which in turn clears standby_cluster from the Patroni DCS and triggers promotion to primary.
Actual behavior
The inner PostgresCluster is not updated. The operator appears to use a merge patch that omits absent fields rather than explicitly nulling them out, so the existing standby field on the inner object persists across reconciles.
Workaround
Manually patch the inner PostgresCluster to remove the standby field:
oc patch postgrescluster.upstream.pgv2.percona.com
-n --type=merge
-p '{"spec":{"standby":null}}'
This clears the DCS standby_cluster config and Patroni promotes to primary within ~30 seconds.
Steps to reproduce
- Create a PerconaPGCluster with spec.standby.enabled: true pointing to a pgBackRest repo
- Confirm the cluster is in standby (pg_is_in_recovery() = true)
- Remove the spec.standby block from the PerconaPGCluster CR and apply it
- Observe that the outer CR reconciles successfully (observedGeneration matches generation)
- Check the inner PostgresCluster:
oc get postgrescluster.upstream.pgv2.percona.com -n -o jsonpath='{.spec.standby}'
→ standby.enabled: true is still present, generation unchanged - Check Patroni DCS:
oc get endpoints -ha-config -n -o jsonpath='{.metadata.annotations.config}' | python3 - m json.tool | grep -A5 standby
→ standby_cluster block is still present in DCS - pg_is_in_recovery() still returns true
- Restarting the operator pod does not resolve the issue
Versions
- OpenShift 4.20.26
- Operator 3.0.0
- Database Postgres 16
Anything else?
Besides this issue, which can be worked around quite effortlessly, so far the migrations from crunchy to this operator have been going well, great work 👍
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the reconciliation that updates the inner PostgresCluster when PerconaPGCluster.spec.standby is removed, focusing on the merge-patch behavior described in the report. Reproduce the listed standby-to-primary steps and verify that the inner standby field and Patroni DCS standby_cluster are cleared and pg_is_in_recovery() becomes false.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, postgresql
- Domain
- databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100