bottlerocket-os / bottlerocket-os/bottlerocket

Pod connectivity issues on Kubernetes 1.34 (EKS) with kube-proxy in `nftables` mode and aws-node NetworkPolicy enabled

Open
#4,674 10 comments 6 reactions 0 assignees View on GitHub
status/needs-triage type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

**Image I'm using:**

- Bottlerocket v1.49.0
- amazon-k8s-cni:v1.20.4-eksbuild.1
- aws-network-policy-agent:v1.2.7-eksbuild.1
- kube-proxy:v1.34.0-eksbuild.4

**What I expected to happen:**

Keep pod connectivity.

**What actually happened:**

After upgrading to Kubernetes 1.34 and restarting nodes, some pods are facing connectivity issues on 5 out of 8 clusters.

For example, coredns sometimes fails to reach apiserver. It fails to start with this error:
```
[ERROR] plugin/kubernetes: Failed to watch on the same host
```

```
kubectl -n kube-system debug -it coredns-67cb468c85-k8txh --image=alpine --target=coredns -- sh
apk add --no-cache curl
curl -k -sS https://172.20.0.1:443/readyz?verbose
[stuck]
```

- When connecting on the node hosting the coredns pod, the apiserver service is reachable.
- bottlerocket, aws-node and kube-proxy properly uses `iptables-nftables` (not the legacy variant)

```
$ kubectl -n kube-system exec aws-node-rl2kp -c aws-node -- iptables -V
iptables v1.8.4 (nf_tables)
```

**Workaround**

- Deleting the pod temporarily fixes the issue (until a new pod has connectivity issues)

To fix the cluster:
- Switch kube-proxy to `iptables` mode fixes the issue on the cluster
- Stick to Kubernetes 1.33
- Disable Network Policy agent of aws-vpc-cni

**How to reproduce the problem:**

Not sure yet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.