[EKS] [request]: restore SNAT functionality disabled since CNI version 1.8
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Per the AWS SNAT documentation, containers egress traffic outside of a local VPC should report the Node IP, not the IP of the pod. However, after speaking with AWS support, this is no longer the case. With an ENIConfig enabled, Pod traffic is always routed through the Pods own ENIs, resulting in the Pod's IP being reported to services in other VPCs (specifically over a NLB Privatelink with Proxy Protocol v2 and preserve_client_ips).
The setting `AWS_EXTERNAL_SERVICE_CIDR` seems to allow this, but it only creates a ip rule for ingress (IE, `
1535: from all to 10.192.0.0/16 lookup main`), and egress traffic is not affected.
Ideally, the original SNAT configuration could be restored via an environment variable to the `aws-node` daemonset, or a list of CIDRs where SNAT is required for egress could be specified via a new environment variable, similar to how `AWS_EXTERNAL_SERVICE_CIDR` works.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We have provisioned a large cluster in a `/18` CIDR (XX.128.0.0/18). Pods on this cluster reach out to customer VPCs via PrivateLink and we are unable to control the NLB setting on the Customer side. Customers have allow-listed the XX.128.0.0/18 CIDR so there are no issues.
We recently exhausted IP space in the original `/18` block, so we adopted a Secondary CIDR via ENIConfig (XX.192.0.0/18). Pods are created with IPs from the Secondary CIDR block, but they are often blocked on the customer side, as the customer has set up allow-lists restricting access to the original CIDR (XX.128.0.0/18)
**Are you currently working around this issue?**
We are avoiding the IP exhaustion issue by manually performing upgrades and draining nodes. We will have to painstakingly work with customer to update their own environments without a resolution to this issue.
Contributor guide
Research direction
Start by reviewing the aws-node DaemonSet configuration and the behavior of AWS_EXTERNAL_SERVICE_CIDR, then examine how ENIConfig and CNI version 1.8 affect egress SNAT. Done means providing a supported way to restore node-IP SNAT or specify egress CIDRs for the affected PrivateLink traffic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100