bottlerocket-os / bottlerocket-os/bottlerocket

kube-proxy iptables mode has 5-6x performance regression on aws-k8s-1.34 due to iptables-nft backend

Open
#4,762 4 comments 4 reactions 0 assignees View on GitHub
status/needs-triage type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

**Image I'm using:**

- Bottlerocket OS 1.54.0 (aws-k8s-1.34) - kernel 6.12.63
- Bottlerocket OS 1.54.0 (aws-k8s-1.32) - kernel 6.1.159 (for comparison)
- Bottlerocket OS 1.54.0 (aws-k8s-1.33) - kernel 6.12.63 (for comparison)

**What I expected to happen:**

kube-proxy running in iptables mode should have similar rule sync performance on aws-k8s-1.34 nodes as on aws-k8s-1.32/1.33 nodes

**What actually happened:**
kube-proxy iptables mode sync latency is 5-6x slower on aws-k8s-1.34 nodes compared to aws-k8s-1.32/1.33 nodes.

| Node Variant | iptables version | Rule Sync Latency | iptables-save time |
|--------------|------------------------------|-------------------|--------------------|
| aws-k8s-1.32 | iptables v1.8.11 (legacy) | 32-52ms | 21ms |
| aws-k8s-1.33 | iptables v1.8.11 (legacy) | 24-34ms | - |
| aws-k8s-1.34 | iptables v1.8.11 (nf_tables) | 172-248ms | 61ms |

Test environment: Same EKS cluster , same kube-proxy version (v1.34.3-eksbuild.2), same workload (877 services, 2057 endpoints).

Root cause: aws-k8s-1.34 switched from legacy iptables to iptables-nft (nftables kernel backend with iptables compatibility layer). Every iptables operation is now translated to nftables, adding significant overhead.

Evidence from the nodes:
# aws-k8s-1.32 (fast)
$ iptables --version
iptables v1.8.11 (legacy)

$ ls -la /usr/sbin/iptables*
# Points to legacy iptables binaries

# aws-k8s-1.34 (slow)
$ iptables --version
iptables v1.8.11 (nf_tables)

$ ls -la /usr/sbin/iptables*
lrwxrwxrwx. 1 root root 7 Jan 23 01:35 /usr/sbin/iptables -> xtables
lrwxrwxrwx. 1 root root 17 Jan 23 01:35 /usr/sbin/iptables-nft -> xtables-nft-multi

**How to reproduce the problem:**

1. Create an EKS cluster with mixed node groups:
- Node group A: Bottlerocket aws-k8s-1.32 AMI
- Node group B: Bottlerocket aws-k8s-1.34 AMI
2. Deploy kube-proxy in iptables mode (EKS default)
3. Deploy enough services to generate meaningful load (100+ services)
4. Check kube-proxy logs on both node types:
kubectl logs -n kube-system | grep "SyncProxyRules complete"

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.