zalando / zalando/postgres-operator
Protect operator CRDs from accidental deletion via metadata finalizer
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
Problem
A kubectl delete crd <name> against the operator's postgresqls, operatorconfigurations, postgresteams or fabriceventstreams CRD is destructive: the Kubernetes apiserver cascades the delete to every custom resource of that kind in the cluster, so a single mistyped command can wipe out every postgresql/operatorconfiguration/PostgresTeam/FabricEventStream object at once. For an operator whose value proposition is the data behind those CRs, this is a one-command data-loss vector.
This is the same class of accidental-cluster-wipe that has been reported across the Kubernetes ecosystem; the standard mitigation is a metadata finalizer on the CRD that blocks deletion until removed explicitly.
Proposed change
Add an acid.zalan.do/crd-protection finalizer to each operator CRD's metadata. With the finalizer set, the apiserver keeps the CRD in Terminating instead of dropping it, so the cascade to existing CRs never happens. Removing the finalizer is the documented manual step required to delete the CRD.
Concretely:
- The Go loader (
PostgresCRD/OperatorConfigurationCRDinpkg/apis/acid.zalan.do/v1/crds.go) sets the finalizer when the CRD object is constructed. This survives thecontroller-genregeneration that rewrites the CRD YAML on everymake, so the finalizer cannot be silently dropped by a regeneration. - The generated CRD YAML manifests (
manifests/,charts/postgres-operator/crds/, the embeddedpkg/apis/.../*.crd.yaml) also carry the finalizer so thatkubectl apply/ Helm installs put it in place directly. - On startup, the operator re-applies the finalizer to the
postgresqlandoperatorconfigurationCRDs if a pre-existing CRD is missing it, so older deployments pick up the safety net on upgrade. - The
postgresteamandfabriceventstreamCRDs are not auto-registered by the operator, so their protection relies on the finalizer shipped in the YAML manifest at install time.
Manual removal procedure
To intentionally delete a CRD, clear the finalizer first:
kubectl patch crd postgresqls.acid.zalan.do -p '{"metadata":{"finalizers":[]}}' --type=merge
kubectl delete crd postgresqls.acid.zalan.do
Per the Kubernetes docs, delete the CRs first when possible, wait for their finalizers to clear, and only then drop the CRDs.
Test
A new e2e test (test_zy_crd_protection_finalizer in e2e/tests/test_e2e.py) targets the most critical CRD - postgresqls.acid.zalan.do - and exercises the full lifecycle:
- Verifies the finalizer is present on every operator CRD.
- Snapshots the live
acid-minimal-clusterpostgresql CR, attempts to delete the CRD, and asserts both the CRD and the CR survive. - Verifies that clearing the finalizer is what actually unblocks the cascade.
- Re-creates the CRD and the cluster in a finally block so a failed assertion cannot leave the suite in a broken state.
The test runs after every other test that depends on the cluster but before the last test_zz_cluster_deletion check.
Out of scope
A unit test for the Go-side ensureCRDProtectionFinalizer helper would be a small addition; the e2e already covers the user-visible behavior end-to-end. Happy to add it if maintainers prefer.
A branch with this change is already pushed to moduon/postgres-operator:crd-protection-finalizer; a PR from there to upstream will follow once this issue gets traction.
Assisted-by: OpenCode + MiniMax M3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pkg/apis/acid.zalan.do/v1/crds.go and compare the generated CRDs under manifests/, charts/postgres-operator/crds/, and pkg/apis/.../*.crd.yaml. Run test_zy_crd_protection_finalizer in e2e/tests/test_e2e.py to understand the lifecycle and expected protection. Done means all four CRDs retain protection, older postgresql and operatorconfiguration CRDs are repaired on startup, and the e2e lifecycle passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, postgresql
- Domain
- databases, infrastructure, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100