aws / aws/aws-network-policy-agent
Proposed node-agent orphan-GC
- Dominant language
- Go
- Stars
- 75
- Forks
- 68
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 10
Description
A self-healing garbage-collection routine in the network-policy-agent that periodically reclaims orphaned eBPF programs/maps whose pod no longer exists on the node — instead of relying solely on the CNI DeletePodNp call to trigger cleanup.
Proposed approach:
- Extend the existing deleted-pods cleanup routine (startDeletedPodsCleanupRoutine, pkg/ebpf/bpf_client.go) with a periodic sweep.
- Enumerate pinned programs under /sys/fs/bpf/globals/aws/programs/*, parse each podIdentifier (reuse utils.GetPodIdentifierFromBPFPinPath, as recoverBPFState does).
- Resolve each podIdentifier to its pods via ingress/egressProgToPodsMap; if no host veth still exists (utils.GetHostVethName → netlink lookup) and it is not still referenced (existing isProgramReferenced
guard), call deleteBPFProbes(podIdentifier).
- Apply a grace period (reuse deletedPodsMinAge) to avoid racing pod ADD.
Why is this needed:
Today DeleteBPFProbes (the only unpin path) is invoked exclusively by the CNI DeletePodNp gRPC. When that call never arrives, the ingress+egress programs stay pinned in bpffs forever, growing unbounded per
node. This happens through no fault of the agent, via several triggers we've confirmed in production:
- Branch-ENI / Security-Groups-for-Pods + short-lived Job pods: ipamd DelNetwork must fetch the pod to release the branch ENI, but the completed Job pod is already gone from the K8s API (Failed to get pod
spec: ... Pod not found) → DEL aborts before network-policy cleanup.
- Node reboot / ipamd cold-start: tmpfs-backed checkpoint (/var/run/aws-node/ipam.json) is wiped → ipamd rebuilds from CRI → datastore: unknown pod on delete of pre-reboot pods.
- Partial attach failure (ingress pinned, egress load fails) on single-use Job podIdentifiers.
A node-side reconciler that keys off "pin present but pod/veth gone" makes the agent self-healing regardless of whether or why the CNI delete ever fires, closing all of these leak paths at once.
Contributor guide
Assessment
This issue has not been assessed yet.