bottlerocket-os / bottlerocket-os/bottlerocket
Pod connectivity issues on Kubernetes 1.34 (EKS) with kube-proxy in `nftables` mode and aws-node NetworkPolicy enabled
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
**Image I'm using:**
- Bottlerocket v1.49.0
- amazon-k8s-cni:v1.20.4-eksbuild.1
- aws-network-policy-agent:v1.2.7-eksbuild.1
- kube-proxy:v1.34.0-eksbuild.4
**What I expected to happen:**
Keep pod connectivity.
**What actually happened:**
After upgrading to Kubernetes 1.34 and restarting nodes, some pods are facing connectivity issues on 5 out of 8 clusters.
For example, coredns sometimes fails to reach apiserver. It fails to start with this error:
```
[ERROR] plugin/kubernetes: Failed to watch on the same host
```
```
kubectl -n kube-system debug -it coredns-67cb468c85-k8txh --image=alpine --target=coredns -- sh
apk add --no-cache curl
curl -k -sS https://172.20.0.1:443/readyz?verbose
[stuck]
```
- When connecting on the node hosting the coredns pod, the apiserver service is reachable.
- bottlerocket, aws-node and kube-proxy properly uses `iptables-nftables` (not the legacy variant)
```
$ kubectl -n kube-system exec aws-node-rl2kp -c aws-node -- iptables -V
iptables v1.8.4 (nf_tables)
```
**Workaround**
- Deleting the pod temporarily fixes the issue (until a new pod has connectivity issues)
To fix the cluster:
- Switch kube-proxy to `iptables` mode fixes the issue on the cluster
- Stick to Kubernetes 1.33
- Disable Network Policy agent of aws-vpc-cni
**How to reproduce the problem:**
Not sure yet
Contributor guide
Assessment
This issue has not been assessed yet.