microsoft / microsoft/retina

operator: memleak with identities of deleted Pods retained in memory

Open
#2,647 1 comment 0 reactions 0 assignees View on GitHub
area/operator
Dominant language
Go
Stars
3.2k
Forks
304
Avg merge
1d 19h
Merged PRs (30d)
78

Description

v1.2.3, hubble control plane, EKS + VPC CNI, 3 nodes / 169 pods.

`retina-operator` RSS grows linearly, 27 → 68 MiB over 15 days, no plateau.

The operator holds one entry per allocated identity in `IdentityManager.labelIdentities` and in the underlying `CachingIdentityAllocator` local keys. It is freed only when a Pod delete event is matched in the store (`handlePodDelete`); anything missed is retained for the process lifetime. Every CronJob Pod allocates a unique identity (`job-name`/`controller-uid` labels) — ~4500/day on this cluster.

Evidence: the operator still syncs identities of Jobs that finished days ago, whose `CiliumIdentity` is long gone:

```
level=warn caller=identitybackend/identity.go:184 msg="Unable update CRD identity information with a reference for this node" error="identity (id:\"5277\",key:\"[... k8s:job-name=nats-backup-29773815]\") does not exist"
```

The `job-name` suffix is minutes-since-epoch: that Job ran 2 days earlier, and the oldest one still referenced is 10 days old. The `CiliumIdentity` CRs themselves are GC'd correctly (252 live).

Contributor guide

Open the contributing guide

Research direction

Start by tracing IdentityManager.labelIdentities and the underlying CachingIdentityAllocator local keys, then inspect handlePodDelete and identitybackend/identity.go around line 184. Reproduce or examine how completed CronJob Pods remain referenced after their CiliumIdentity CRs are gone; done means stale identities are released without disrupting active identity synchronization.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.