zalando / zalando/postgres-operator

EndpointSlices not cleaned up after replica service deletion (orphaned resources)

Open
#2,973 0 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Which image of the operator are you using? 1.14.0
Where do you run it - cloud or metal? Vanilla K8S/Bare metall
Are you running Postgres Operator in production? Yes
Type of issue? Bug report

Problem Description
We're experiencing a resource leak where EndpointSlices for -repl services are not being cleaned up after the parent Service is deleted. This has triggered quota alerts in our namespace.

Observed Behavior
When a PostgreSQL cluster is deleted, the operator properly deletes the replica service (*-repl)
However, the associated EndpointSlices remain in the namespace as orphaned resources
This only affects -repl services - other services (-config, master endpoint) clean up properly
The EndpointSlices have proper ownerReferences pointing to the Service:

  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Service
    name: endpoints-keys-test-repl
    uid: 4d904c2d-cfff-4ee1-bd50-3215dc16de82

Reproduction Steps

Create a Spilo PostgreSQL cluster
Delete the cluster
Check for remaining EndpointSlices:

kubectl get endpointslices -n <namespace> | grep db-repl

Orphaned EndpointSlices will remain despite parent Services being deleted

Expected Behavior
EndpointSlices should be automatically cleaned up by Kubernetes garbage collector when the parent Service is deleted (via ownerReferences).

Investigation Findings
The operator does not create an endpoint for -repl during cluster creation, only the service.
However, at the moment of cluster deletion attempts to clean this object.
From operator logs during deletion:

time="2025-10-30T11:26:09Z" level=debug msg="deleting replica endpoint" cluster-name=service-keys/endpoints-keys-test pkg=cluster worker=3
time="2025-10-30T11:26:09Z" level=info msg="replica endpoint \"service-keys/endpoints-keys-test-repl\" has been deleted" cluster-name=service-keys/endpoints-keys-test pkg=cluster worker=3
time="2025-10-30T11:26:09Z" level=debug msg="deleting replica service" cluster-name=service-keys/endpoints-keys-test pkg=cluster worker=3
time="2025-10-30T11:26:09Z" level=info msg="replica service \"service-keys/endpoints-keys-test-repl\" has been deleted" cluster-name=service-keys/endpoints-keys-test pkg=cluster worker=3

From Kubernetes API logs, the garbage collector attempts to patch the EndpointSlice but doesn't delete it:

Oct 30, 2025 @ 11:26:09.840  /apis/discovery.k8s.io/v1/namespaces/service-keys/endpointslices/endpoints-keys-test-repl-9p7pf
ResponseComplete  patch  
system:serviceaccount:kube-system:generic-garbage-collector

Hypothesis
It appears the operator may be deleting the Service with orphan propagation policy instead of background/foreground, which would explain why the garbage collector only patches but never deletes the EndpointSlices.

Impact
Resource quota exhaustion in namespaces with many demo/test databases
Manual cleanup required periodically
Affects all -repl services across multiple clusters

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing cluster deletion and inspecting the replica service and its EndpointSlices with kubectl. Trace the operator's replica service deletion path and compare the ownerReferences and propagation behavior with the services that clean up correctly. Done means deleting a cluster removes the -repl Service and leaves no orphaned EndpointSlices.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
backend, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.