operator: memleak with identities of deleted Pods retained in memory
- Dominant language
- Go
- Stars
- 3.2k
- Forks
- 304
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 78
Description
v1.2.3, hubble control plane, EKS + VPC CNI, 3 nodes / 169 pods.
`retina-operator` RSS grows linearly, 27 → 68 MiB over 15 days, no plateau.
The operator holds one entry per allocated identity in `IdentityManager.labelIdentities` and in the underlying `CachingIdentityAllocator` local keys. It is freed only when a Pod delete event is matched in the store (`handlePodDelete`); anything missed is retained for the process lifetime. Every CronJob Pod allocates a unique identity (`job-name`/`controller-uid` labels) — ~4500/day on this cluster.
Evidence: the operator still syncs identities of Jobs that finished days ago, whose `CiliumIdentity` is long gone:
```
level=warn caller=identitybackend/identity.go:184 msg="Unable update CRD identity information with a reference for this node" error="identity (id:\"5277\",key:\"[... k8s:job-name=nats-backup-29773815]\") does not exist"
```
The `job-name` suffix is minutes-since-epoch: that Job ran 2 days earlier, and the oldest one still referenced is 10 days old. The `CiliumIdentity` CRs themselves are GC'd correctly (252 live).
Contributor guide
Research direction
Start by tracing IdentityManager.labelIdentities and the underlying CachingIdentityAllocator local keys, then inspect handlePodDelete and identitybackend/identity.go around line 184. Reproduce or examine how completed CronJob Pods remain referenced after their CiliumIdentity CRs are gone; done means stale identities are released without disrupting active identity synchronization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100