aws / aws/aws-network-policy-agent
Long delays to accept new ingress flows from new pods
- Dominant language
- Go
- Stars
- 76
- Forks
- 69
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 10
Description
**What happened**:
We are observing an issue in our EKS clusters where new client pod traffic is denied on the receiving side for a period (ranging from a few seconds to several minutes), until the flow is accepted. This is causing many spikes of tail latency across many production services. When we redeploy the client pod, we see `Verdict DENY` flow logs from the new pod for some time before finally switching to `Verdict ACCEPT`. Here is an example with nearly 3-minute delay, the new client pod is `10.5.223.87` and we see these messages in the `network-policy-agent.log` file on the receiving pod's host:
```
{"level":"info","ts":"2025-11-21T03:24:44.553Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56646 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:24:45.586Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56646 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:24:47.676Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56646 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:24:51.746Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56646 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:24:57.581Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 50600 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:24:58.626Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 50600 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:00.706Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 50600 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:04.786Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 50600 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:13.645Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 46056 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:14.706Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 46056 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:16.636Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 55760 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:16.786Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 46056 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:17.666Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 55760 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:19.746Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 55760 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:20.866Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 46056 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:23.826Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 55760 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:35.697Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42802 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:36.708Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42802 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:38.786Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42802 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:41.724Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42808 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:42.786Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42808 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:42.866Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42802 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:44.866Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42808 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:48.946Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42808 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:54.751Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 59622 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:55.826Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 59622 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:57.906Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 59622 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:01.996Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 59622 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:13.806Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56272 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:14.866Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56272 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:16.946Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56272 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:21.026Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56272 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:42.890Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56878 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:43.906Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56878 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:45.986Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56878 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:50.076Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56878 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:57.958Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41482 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:59.026Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41482 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:01.106Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41482 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:03.968Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41486 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:05.026Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41486 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:05.192Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41482 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:07.106Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41486 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:11.186Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41486 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:23.011Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35068 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:24.076Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35068 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:26.146Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35068 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:30.026Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35908 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:30.226Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35068 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:31.106Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35908 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"debug","ts":"2025-11-21T03:27:31.931Z","caller":"ebpf/bpf_client.go:1074","msg":"No L4 specified. Add Catch all entry CIDR: 10.5.223.87/32"}
{"level":"info","ts":"2025-11-21T03:27:31.939Z","caller":"ebpf/bpf_client.go:1074","msg":"Updating Map with IP Key: 10.5.223.87/32"}
{"level":"debug","ts":"2025-11-21T03:27:33.186Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35908 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict ACCEPT Direction ingress"}
{"level":"debug","ts":"2025-11-21T03:27:35.259Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 38164 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict ACCEPT Direction ingress"}
{"level":"debug","ts":"2025-11-21T03:27:35.310Z","caller":"ebpf/bpf_client.go:1074","msg":"No L4 specified. Add Catch all entry CIDR: 10.5.223.87/32"}
{"level":"info","ts":"2025-11-21T03:27:35.318Z","caller":"ebpf/bpf_client.go:1074","msg":"Updating Map with IP Key: 10.5.223.87/32"}
{"level":"debug","ts":"2025-11-21T03:27:44.470Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41508 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict ACCEPT Direction ingress"}
{"level":"debug","ts":"2025-11-21T03:27:46.533Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41520 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict ACCEPT Direction ingress"}
```
**Attach logs**
Logs sent to `k8s-awscni-triage@amazon.com`
**What you expected to happen**:
Short or negligible delays for a new flows from new client pods to be accepted on the ingress side when NetworkPolicy allows the traffic on both sides.
**How to reproduce it (as minimally and precisely as possible)**:
We have been able to reproduce this at will in our large clusters (700 nodes, 11000 pods). We have two pods, a client and a server, in different namespaces running on different nodes with NetworkPolicy resources on both sides. Restarting the client pod nearly always triggers the issue.
**Anything else we need to know?**:
We are using the default `NETWORK_POLICY_ENFORCING_MODE=standard`
**Environment**:
- Kubernetes version (use `kubectl version`): `Server Version: v1.33.5-eks-3cfe0ce`
- CNI Version: `v1.20.4-eksbuild.1`
- Network Policy Agent Version: `v1.2.7`
- OS (e.g: `cat /etc/os-release`): `Amazon Linux 2023.8.20250707`
- Kernel (e.g. `uname -a`): `6.1.141-165.249.amzn2023.aarch64 #1 SMP Tue Jul 1 18:00:46 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux`
Contributor guide
Assessment
This issue has not been assessed yet.