percona / percona/percona-postgresql-operator

PerconaPGCluster standby removal not propagated to inner PostgresCluster — cluster stuck in standby after spec.standby removed

Open
#1,687 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug jira-task
Dominant language
Go
Stars
385
Forks
83
Avg merge
2d 5h
Merged PRs (30d)
39

Description

Report

When a PerconaPGCluster is configured with spec.standby and then the spec.standby block is removed (to promote the cluster to primary), the Percona operator reconciles the outer CR but does not update the inner upstream.pgv2.percona.com/v1beta1 PostgresCluster object it manages. The inner object retains spec.standby.enabled: true, which keeps the Patroni DCS standby_cluster config in place and leaves PostgreSQL in recovery mode indefinitely.

More about the problem

Expected behavior

Removing spec.standby from PerconaPGCluster should cause the operator to update the inner PostgresCluster to remove spec.standby, which in turn clears standby_cluster from the Patroni DCS and triggers promotion to primary.

Actual behavior

The inner PostgresCluster is not updated. The operator appears to use a merge patch that omits absent fields rather than explicitly nulling them out, so the existing standby field on the inner object persists across reconciles.

Workaround

Manually patch the inner PostgresCluster to remove the standby field:

oc patch postgrescluster.upstream.pgv2.percona.com
-n --type=merge
-p '{"spec":{"standby":null}}'
This clears the DCS standby_cluster config and Patroni promotes to primary within ~30 seconds.

Steps to reproduce
  1. Create a PerconaPGCluster with spec.standby.enabled: true pointing to a pgBackRest repo
  2. Confirm the cluster is in standby (pg_is_in_recovery() = true)
  3. Remove the spec.standby block from the PerconaPGCluster CR and apply it
  4. Observe that the outer CR reconciles successfully (observedGeneration matches generation)
  5. Check the inner PostgresCluster:
    oc get postgrescluster.upstream.pgv2.percona.com -n -o jsonpath='{.spec.standby}'
    → standby.enabled: true is still present, generation unchanged
  6. Check Patroni DCS:
    oc get endpoints -ha-config -n -o jsonpath='{.metadata.annotations.config}' | python3 - m json.tool | grep -A5 standby
    → standby_cluster block is still present in DCS
  7. pg_is_in_recovery() still returns true
  8. Restarting the operator pod does not resolve the issue
Versions
  1. OpenShift 4.20.26
  2. Operator 3.0.0
  3. Database Postgres 16
Anything else?

Besides this issue, which can be worked around quite effortlessly, so far the migrations from crunchy to this operator have been going well, great work 👍

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the reconciliation that updates the inner PostgresCluster when PerconaPGCluster.spec.standby is removed, focusing on the merge-patch behavior described in the report. Reproduce the listed standby-to-primary steps and verify that the inner standby field and Patroni DCS standby_cluster are cleared and pg_is_in_recovery() becomes false.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.