zalando / zalando/postgres-operator

Protect operator CRDs from accidental deletion via metadata finalizer

Open
#3,168 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Problem

A kubectl delete crd <name> against the operator's postgresqls, operatorconfigurations, postgresteams or fabriceventstreams CRD is destructive: the Kubernetes apiserver cascades the delete to every custom resource of that kind in the cluster, so a single mistyped command can wipe out every postgresql/operatorconfiguration/PostgresTeam/FabricEventStream object at once. For an operator whose value proposition is the data behind those CRs, this is a one-command data-loss vector.

This is the same class of accidental-cluster-wipe that has been reported across the Kubernetes ecosystem; the standard mitigation is a metadata finalizer on the CRD that blocks deletion until removed explicitly.

Proposed change

Add an acid.zalan.do/crd-protection finalizer to each operator CRD's metadata. With the finalizer set, the apiserver keeps the CRD in Terminating instead of dropping it, so the cascade to existing CRs never happens. Removing the finalizer is the documented manual step required to delete the CRD.

Concretely:

  • The Go loader (PostgresCRD / OperatorConfigurationCRD in pkg/apis/acid.zalan.do/v1/crds.go) sets the finalizer when the CRD object is constructed. This survives the controller-gen regeneration that rewrites the CRD YAML on every make, so the finalizer cannot be silently dropped by a regeneration.
  • The generated CRD YAML manifests (manifests/, charts/postgres-operator/crds/, the embedded pkg/apis/.../*.crd.yaml) also carry the finalizer so that kubectl apply / Helm installs put it in place directly.
  • On startup, the operator re-applies the finalizer to the postgresql and operatorconfiguration CRDs if a pre-existing CRD is missing it, so older deployments pick up the safety net on upgrade.
  • The postgresteam and fabriceventstream CRDs are not auto-registered by the operator, so their protection relies on the finalizer shipped in the YAML manifest at install time.

Manual removal procedure

To intentionally delete a CRD, clear the finalizer first:

kubectl patch crd postgresqls.acid.zalan.do -p '{"metadata":{"finalizers":[]}}' --type=merge
kubectl delete crd postgresqls.acid.zalan.do

Per the Kubernetes docs, delete the CRs first when possible, wait for their finalizers to clear, and only then drop the CRDs.

Test

A new e2e test (test_zy_crd_protection_finalizer in e2e/tests/test_e2e.py) targets the most critical CRD - postgresqls.acid.zalan.do - and exercises the full lifecycle:

  • Verifies the finalizer is present on every operator CRD.
  • Snapshots the live acid-minimal-cluster postgresql CR, attempts to delete the CRD, and asserts both the CRD and the CR survive.
  • Verifies that clearing the finalizer is what actually unblocks the cascade.
  • Re-creates the CRD and the cluster in a finally block so a failed assertion cannot leave the suite in a broken state.

The test runs after every other test that depends on the cluster but before the last test_zz_cluster_deletion check.

Out of scope

A unit test for the Go-side ensureCRDProtectionFinalizer helper would be a small addition; the e2e already covers the user-visible behavior end-to-end. Happy to add it if maintainers prefer.

A branch with this change is already pushed to moduon/postgres-operator:crd-protection-finalizer; a PR from there to upstream will follow once this issue gets traction.

Assisted-by: OpenCode + MiniMax M3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pkg/apis/acid.zalan.do/v1/crds.go and compare the generated CRDs under manifests/, charts/postgres-operator/crds/, and pkg/apis/.../*.crd.yaml. Run test_zy_crd_protection_finalizer in e2e/tests/test_e2e.py to understand the lifecycle and expected protection. Done means all four CRDs retain protection, older postgresql and operatorconfiguration CRDs are repaired on startup, and the e2e lifecycle passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, infrastructure, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.