aws / aws/aws-network-policy-agent
Memory leak: podIdentifierToPolicyEndpointMap inserts entries for all pods cluster-wide but only removes node-local ones
- Dominant language
- Go
- Stars
- 75
- Forks
- 68
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 10
Description
**What happened**:
On a ~1,000-node cluster with high pod churn (~150 pod creates+deletes/s, uniquely-named pods) under 3 long-lived egress NetworkPolicies (~45 PolicyEndpoint slices, ~35k selected pods), every aws-eks-nodeagent leaks Go heap at ~420 MiB/h until OOM-killed (~every 10h at a 4Gi limit), fleet-wide. Each restart causes a window of `plugin type="aws-cni" failed (add): add cmd: failed to setup network policy` pod-create failures.
Growth is identical on nodes hosting zero policy-selected pods. go_memstats_heap_inuse_bytes tracks container age linearly (608Mi @ 1.9h → 1342Mi @ 4.5h → 1570Mi @ 4.8h; RSS ≈ heap). A read-only scan of one agent's live heap (800MiB sample via /proc//mem) found ~600k copies of our pod-name prefix (~17× the number of live pods) and ~54k copies of a single NetworkPolicy's name (retained parentPEList slices) i.e. per-pod bookkeeping for pods that no longer exist.
Note that pods must be uniquely-named and names remain unique after GetPodIdentifier strips the final dash-segment, e.g. `job-sb-019f3e0d-441f-…-` → identifier `job-sb-019f3e0d-441f-…` (one per pod).
Root cause appears visible in v1.3.5 controllers/policyendpoints_controller.go:
- Insertion is cluster-wide: deriveTargetPods calls updatePodIdentifierToPEMap(podIdentifier, parentPEList) at [L622](https://github.com/aws/aws-network-policy-agent/blob/v1.3.5/controllers/policyendpoints_controller.go#L622), outside the nodeIP.Equal(pod.HostIP) check at L617 — so every agent stores an entry for every selected pod in the cluster.
- Removal is local-only: targetPodIdentifiers is populated only inside the node-IP match (L617–621); [deriveStalePodIdentifiers (L680)](https://github.com/aws/aws-network-policy-agent/blob/v1.3.5/controllers/policyendpoints_controller.go#L680-L700) computes staleness against that node-local set only.
- `cleanUpPolicyEndpoint` runs only on PE deletion, which never happens while the parent NetworkPolicy exists (pod deletion = slice update).
So identifiers for non-local pods (≈ (N−1)/N of entries on an N-node cluster) have no deletion path while the policy lives. With uniquely-named pods, every pod ever created adds a permanent map entry on every node. Deployments largely dodge this because GetPodIdentifier collapses names per ReplicaSet — which may be why it isn't reported more widely (possibly related: [#213](https://github.com/aws/aws-network-policy-agent/issues/213)).
**Attach logs**
Can send the log archive on request; logs are not the interesting artifact here — heap metrics and heap contents above are.
**What you expected to happen**: Agent memory bounded by the number of live policy-selected pods (ideally node-local ones), regardless of cumulative churn.
**How to reproduce it (as minimally and precisely as possible)**:
1. Create a NetworkPolicy selecting a namespace; keep ≥1 matching pod alive so its PolicyEndpoints are never deleted.
2. Churn pods whose names remain unique after stripping the final dash-segment e.g. a fresh Job per task with a UUID in the Job name (pods `task--` → identifier `task--`). Pods of a single long-lived Job/Deployment will not reproduce: GetPodIdentifier collapses them to one identifier.
3. Watch go_memstats_heap_inuse_bytes on the agent of any node including nodes never hosting these pods. It grows without bound.
**Anything else we need to know?**:
Likely the same family as [#444](https://github.com/aws/aws-network-policy-agent/issues/444) (symptom-level report, closed without RCA) and [#213](https://github.com/aws/aws-network-policy-agent/issues/213).
**Environment**:
- Kubernetes version: v1.35.6-eks-8f14419
- CNI Version: v1.22.1-eksbuild.2 (EKS addon)
- Network Policy Agent Version: v1.3.5-eksbuild.3
- OS: Amazon Linux 2023.10.20260216
- Kernel: 6.12.68-92.122.amzn2023.x86_64
Contributor guide
Assessment
This issue has not been assessed yet.