[EKS] AZ-isolated control-plane connectivity: X-ENI per registered subnet + AZ-affinity routing
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Provide AZ-isolated control-plane connectivity for multi-AZ EKS clusters. Concretely:
1. **Placement**: place one cross-account ENI (X-ENI) in every registered cluster subnet, so a 3-AZ cluster has 3 X-ENIs (one per AZ), instead of the current fixed 2.
2. **AZ-affinity routing** (both directions, by default):
- Node → apiserver: populate the `default/kubernetes` Service endpoints with topology hints so kube-proxy / CNIs honoring topology aware routing prefer the same-AZ X-ENI.
- Apiserver → pod: when the apiserver initiates a connection to a pod (admission webhook, APIService aggregation, `kubectl exec/portforward`, readiness-probe forwarding), egress via the X-ENI in the target pod's AZ.
Together, (1) and (2) deliver the property that **a network failure isolated to a single AZ only affects that AZ's data plane**, not the whole cluster.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
EKS currently always provisions exactly 2 Control Plane Instances (CPIs) and exactly 2 X-ENIs per cluster, regardless of how many subnets are registered, with X-ENI AZ placement chosen randomly by the CPI auto scaling group from the registered subnets. (Confirmed via AWS Support case `case-211125670924-muen-2026-19f3497cc2455bf7`.) For a cluster registered across 3 AZs, exactly one AZ is always without an X-ENI.
This forces a **structural ~50% blast radius for any single-AZ network failure** affecting whichever AZ hosts one of the 2 X-ENIs:
- Both `node → apiserver` traffic (kubelet, kube-proxy, controllers) and `apiserver → pod` traffic (admission webhooks, APIService aggregation, `kubectl exec/portforward`) are flow-hashed across the 2 X-ENIs.
- Flow-hashed selection is independent of source/destination AZ. Losing 1 X-ENI therefore disrupts ~50% of all flows in both directions across the **entire cluster**, not just within the failing AZ.
We observed this end-to-end during a controlled fault-injection exercise (subnet-level traffic block in one AZ on a 3-AZ EKS 1.33 cluster):
- ~50% of in-cluster admission webhook calls timed out at the apiserver
- Apiserver request handlers saturated, hit `http: Handler timeout` (60s)
- Apiserver watch cache backpressured and force-closed long-lived watch streams (`cache_watcher.go:203] Forcing ... watcher close due to unresponsiveness ... graceful = false`)
- Kubelets across **all 3 AZs** experienced `http2: client connection lost` and failed lease renewals — leading to NodeNotReady flaps on nodes nowhere near the failing AZ
- Cluster-wide `TaintManagerEviction` cascade
The desired property is **AZ isolation**: a network failure scoped to a single AZ should affect only that AZ's data plane, with the rest of the cluster (data plane in other AZs + control plane) continuing to operate normally. This is the same isolation property that customers already expect from cross-zone-off NLBs, topology-aware EndpointSlices, and the cell-based EKS pattern AWS recommends in [Cell-Based Architecture for High Availability](https://aws.amazon.com/solutions/guidance/cell-based-architecture-for-amazon-eks/) — but achieved as a property of a single multi-AZ cluster rather than requiring fleets of single-AZ clusters.
Crucially, more X-ENIs alone are not sufficient. With 3 X-ENIs but unchanged flow-hashing, an AZ failure still scatters ~33% of flows cluster-wide rather than being contained. The AZ-affinity routing piece (item 2 above) is what converts "more X-ENIs" into actual isolation.
**Are you currently working around this issue?**
Customer-side mitigations are limited and partial:
- Tightening admission-webhook `timeoutSeconds` (e.g. Kyverno) reduces apiserver handler-saturation amplitude.
- Spreading critical in-cluster services (metrics-server, admission webhooks) across multiple AZs.
- Topology-aware routing / `trafficDistribution: PreferClose` for in-cluster Services (does not affect the `default/kubernetes` Service nor the apiserver's egress path to pods).
- Cell-based architecture (one EKS cluster per AZ) — the only architectural path AWS Support endorsed for true single-AZ blast immunity, but operationally significant: N× cluster proliferation, cross-cluster service discovery, multi-cluster ArgoCD/GitOps overhead, and per-cluster fixed-cost overhead.
None of these mitigate the structural 50% control-plane reduction during a single-AZ failure on a 3-AZ cluster. AWS Support confirmed there is currently no supported mechanism customer-side to influence X-ENI count, X-ENI placement, or apiserver egress AZ affinity, and that the CPI count is fixed at 2.
**Additional context**
Per AWS Support: X-ENIs are bound 1:1 with Control Plane Instances; the AZ choice is made by the CPI ASG randomly from the registered subnets; `UpdateClusterConfig` only changes the candidate subnet pool but does not trigger new X-ENIs; new X-ENIs are only created on instance refresh, with the new ASG-selected AZ.
Comparable isolation features already exist in adjacent AWS services and Kubernetes:
- NLB cross-zone load balancing off → AZ-isolated client-to-target traffic.
- Kubernetes [topology-aware routing / hints](https://kubernetes.io/docs/concepts/services-networking/topology-aware-routing/) and `trafficDistribution: PreferClose` → AZ-affinity for application Services.
- ARC zonal shift / zonal autoshift on EKS — useful for shifting workload traffic, but does not address X-ENI placement or apiserver egress affinity.
A configurable AZ-isolated mode for managed EKS would let customers pick the resilience trade-off without inverting their cluster topology.
**Attachments**
n/a
Contributor guide
Research direction
Start by reviewing the issue's two requirements: one X-ENI per registered subnet and AZ-affinity for node-to-apiserver and apiserver-to-pod traffic. Read the linked Kubernetes topology-aware routing and AWS cell-based architecture material, then determine how a configurable EKS mode could provide single-AZ isolation; done means both traffic directions remain contained to the affected AZ.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100