aws / aws/aws-network-policy-agent

Memory leak: podIdentifierToPolicyEndpointMap inserts entries for all pods cluster-wide but only removes node-local ones

Open
#613 0 comments 0 reactions 1 assignee Claimed by @jaydeokar View on GitHub
bug
Dominant language
Go
Stars
75
Forks
68
Avg merge
2d 11h
Merged PRs (30d)
10

Description

**What happened**:

On a ~1,000-node cluster with high pod churn (~150 pod creates+deletes/s, uniquely-named pods) under 3 long-lived egress NetworkPolicies (~45 PolicyEndpoint slices, ~35k selected pods), every aws-eks-nodeagent leaks Go heap at ~420 MiB/h until OOM-killed (~every 10h at a 4Gi limit), fleet-wide. Each restart causes a window of `plugin type="aws-cni" failed (add): add cmd: failed to setup network policy` pod-create failures.

Growth is identical on nodes hosting zero policy-selected pods. go_memstats_heap_inuse_bytes tracks container age linearly (608Mi @ 1.9h → 1342Mi @ 4.5h → 1570Mi @ 4.8h; RSS ≈ heap). A read-only scan of one agent's live heap (800MiB sample via /proc//mem) found ~600k copies of our pod-name prefix (~17× the number of live pods) and ~54k copies of a single NetworkPolicy's name (retained parentPEList slices) i.e. per-pod bookkeeping for pods that no longer exist.

Note that pods must be uniquely-named and names remain unique after GetPodIdentifier strips the final dash-segment, e.g. `job-sb-019f3e0d-441f-…-` → identifier `job-sb-019f3e0d-441f-…` (one per pod).

Root cause appears visible in v1.3.5 controllers/policyendpoints_controller.go:

- Insertion is cluster-wide: deriveTargetPods calls updatePodIdentifierToPEMap(podIdentifier, parentPEList) at [L622](https://github.com/aws/aws-network-policy-agent/blob/v1.3.5/controllers/policyendpoints_controller.go#L622), outside the nodeIP.Equal(pod.HostIP) check at L617 — so every agent stores an entry for every selected pod in the cluster.
- Removal is local-only: targetPodIdentifiers is populated only inside the node-IP match (L617–621); [deriveStalePodIdentifiers (L680)](https://github.com/aws/aws-network-policy-agent/blob/v1.3.5/controllers/policyendpoints_controller.go#L680-L700) computes staleness against that node-local set only.
- `cleanUpPolicyEndpoint` runs only on PE deletion, which never happens while the parent NetworkPolicy exists (pod deletion = slice update).

So identifiers for non-local pods (≈ (N−1)/N of entries on an N-node cluster) have no deletion path while the policy lives. With uniquely-named pods, every pod ever created adds a permanent map entry on every node. Deployments largely dodge this because GetPodIdentifier collapses names per ReplicaSet — which may be why it isn't reported more widely (possibly related: [#213](https://github.com/aws/aws-network-policy-agent/issues/213)).

**Attach logs**

Can send the log archive on request; logs are not the interesting artifact here — heap metrics and heap contents above are.

**What you expected to happen**: Agent memory bounded by the number of live policy-selected pods (ideally node-local ones), regardless of cumulative churn.

**How to reproduce it (as minimally and precisely as possible)**:

1. Create a NetworkPolicy selecting a namespace; keep ≥1 matching pod alive so its PolicyEndpoints are never deleted.
2. Churn pods whose names remain unique after stripping the final dash-segment e.g. a fresh Job per task with a UUID in the Job name (pods `task--` → identifier `task--`). Pods of a single long-lived Job/Deployment will not reproduce: GetPodIdentifier collapses them to one identifier.
3. Watch go_memstats_heap_inuse_bytes on the agent of any node including nodes never hosting these pods. It grows without bound.

**Anything else we need to know?**:

Likely the same family as [#444](https://github.com/aws/aws-network-policy-agent/issues/444) (symptom-level report, closed without RCA) and [#213](https://github.com/aws/aws-network-policy-agent/issues/213).

**Environment**:
- Kubernetes version: v1.35.6-eks-8f14419
- CNI Version: v1.22.1-eksbuild.2 (EKS addon)
- Network Policy Agent Version: v1.3.5-eksbuild.3
- OS: Amazon Linux 2023.10.20260216
- Kernel: 6.12.68-92.122.amzn2023.x86_64

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.