aws / aws/aws-network-policy-agent

Stale PolicyEndpoint when NetworkPolicy podSelector changes, agent keeps enforcing old selector

Open
#643 0 comments 0 reactions 1 assignee Claimed by @jaydeokar View on GitHub
Dominant language
Go
Stars
75
Forks
68
Avg merge
2d 11h
Merged PRs (30d)
10

Description

## Description

When the `podSelector` of an existing NetworkPolicy is updated (e.g. swapped to a value that no longer matches any pod, in order to neutralize the policy), the corresponding `PolicyEndpoint` CRD (`networking.k8s.aws/v1alpha1`) is not reconciled. The PolicyEndpoint stays with the previous selector and `podSelectorEndpoints` list, and the agent keeps enforcing the old rules at the dataplane level.

The PolicyEndpoint only reconciles back to the correct state once it is manually deleted (the controller then recreates it from the up-to-date NetworkPolicy spec).

## Environment

- EKS version: 1.34
- EKS Auto Mode: yes (cluster is on Auto Mode, built-in VPC CNI / network-policy controller)
- aws-network-policy-agent: managed via Auto Mode (version not directly visible)
- VPC CNI addon: managed

## Reproduce

1. Create a NetworkPolicy `np-foo` with `podSelector.matchLabels: {app: bar}` and any egress rule restricting traffic. Confirm pods labeled `app=bar` are enforced (DNS works, etc.).
2. Edit `np-foo` to change `podSelector.matchLabels` to a value that matches no pod, e.g. `{app: disabled-pending-redesign}`. Apply.
3. Wait. Inspect the matching PolicyEndpoint:

```
kubectl get policyendpoints.networking.k8s.aws -n -o yaml
```

## Expected

The PolicyEndpoint reconciles to either:
- An empty `podSelectorEndpoints` (since the new selector matches no pod), or
- It is removed entirely.

## Actual

The PolicyEndpoint keeps the *original* `podSelector` and `podSelectorEndpoints`. The agent continues enforcing the old rules on the listed pods, causing things like DNS egress to be denied even though the source NetworkPolicy no longer applies to them.

In one occurrence, the PolicyEndpoint had a `creationTimestamp` ~29h old while the source NetworkPolicy had been updated multiple hours before — clear evidence the controller never reconciled.

## Workaround

Manually delete the stale PolicyEndpoint:

```
kubectl delete policyendpoints.networking.k8s.aws -n
```

The controller (owner=NetworkPolicy) recreates a fresh PolicyEndpoint that reflects the current selector. Pods recover after restarting them (e.g. `kubectl delete pod`).

For users who actually want to disable network policies entirely on a cluster, the wider cleanup procedure is:
1. Set `enableNetworkPolicy: "false"` on the AWS VPC CNI addon (via EKS console / awscli) with "Override" conflict resolution
2. Delete every NetworkPolicy and every PolicyEndpoint cluster-wide
3. Replace the nodes to clear stale eBPF state maintained by the agent

The selector-swap scenario described above does not need this full procedure — only deleting the stale PolicyEndpoint is enough.

## Related

- aws/aws-network-policy-agent#271 (Network Policy Not Enforced on Initial Creation) — opposite symptom (under-enforcement on creation), but same surface area.
- aws/aws-network-policy-agent#206 (Pod eBPF Map Differs from Policy Endpoint) — adjacent: dataplane state diverging from PolicyEndpoint.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.