bottlerocket-os / bottlerocket-os/bottlerocket
kube-proxy iptables mode has 5-6x performance regression on aws-k8s-1.34 due to iptables-nft backend
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
**Image I'm using:**
- Bottlerocket OS 1.54.0 (aws-k8s-1.34) - kernel 6.12.63
- Bottlerocket OS 1.54.0 (aws-k8s-1.32) - kernel 6.1.159 (for comparison)
- Bottlerocket OS 1.54.0 (aws-k8s-1.33) - kernel 6.12.63 (for comparison)
**What I expected to happen:**
kube-proxy running in iptables mode should have similar rule sync performance on aws-k8s-1.34 nodes as on aws-k8s-1.32/1.33 nodes
**What actually happened:**
kube-proxy iptables mode sync latency is 5-6x slower on aws-k8s-1.34 nodes compared to aws-k8s-1.32/1.33 nodes.
| Node Variant | iptables version | Rule Sync Latency | iptables-save time |
|--------------|------------------------------|-------------------|--------------------|
| aws-k8s-1.32 | iptables v1.8.11 (legacy) | 32-52ms | 21ms |
| aws-k8s-1.33 | iptables v1.8.11 (legacy) | 24-34ms | - |
| aws-k8s-1.34 | iptables v1.8.11 (nf_tables) | 172-248ms | 61ms |
Test environment: Same EKS cluster , same kube-proxy version (v1.34.3-eksbuild.2), same workload (877 services, 2057 endpoints).
Root cause: aws-k8s-1.34 switched from legacy iptables to iptables-nft (nftables kernel backend with iptables compatibility layer). Every iptables operation is now translated to nftables, adding significant overhead.
Evidence from the nodes:
# aws-k8s-1.32 (fast)
$ iptables --version
iptables v1.8.11 (legacy)
$ ls -la /usr/sbin/iptables*
# Points to legacy iptables binaries
# aws-k8s-1.34 (slow)
$ iptables --version
iptables v1.8.11 (nf_tables)
$ ls -la /usr/sbin/iptables*
lrwxrwxrwx. 1 root root 7 Jan 23 01:35 /usr/sbin/iptables -> xtables
lrwxrwxrwx. 1 root root 17 Jan 23 01:35 /usr/sbin/iptables-nft -> xtables-nft-multi
**How to reproduce the problem:**
1. Create an EKS cluster with mixed node groups:
- Node group A: Bottlerocket aws-k8s-1.32 AMI
- Node group B: Bottlerocket aws-k8s-1.34 AMI
2. Deploy kube-proxy in iptables mode (EKS default)
3. Deploy enough services to generate meaningful load (100+ services)
4. Check kube-proxy logs on both node types:
kubectl logs -n kube-system | grep "SyncProxyRules complete"
Contributor guide
Assessment
This issue has not been assessed yet.