[EKS] [request]: ability to set dns resolving for EKS.
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a π [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
It'll be good to set dns resolver for EKS ControlPlane.
It could be any DNS server reachable in VPC (e.g. CoreDNS running on hostNetwork on workload)
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
There is well know issue with webhooks in EKS for non VPC CNI, e.g
https://github.com/cilium/cilium/issues/29454
https://github.com/cilium/cilium/issues/30111
https://github.com/cilium/cilium/issues/21959
EKS Control-plane unable to call webhooks running in overlay network. There is no communication between EKS CP and cilium PodCIDR.
I have idea to implement approach to fix this.
If `enable-aggregator-routing` set to false on api-server, all webhook service reference could be resolved by internal names like:
http://..kubernetes.svc.cluster.local
I want to intercept this dns query on the side of coreDNS (with plugin) running in k8s:
https://github.com/CharlieR-o-o-t/coredns_eks/tree/master/plugin/eksintercept
Then, using DNS, the traffic is routed to a dedicated webhook gateway service endpoint whose pods run with hostNetwork enabled.
The gateway listens on all webhook ports, performs SSL passthrough, supports multiple webhook configurations on the same port, and forwards L3 traffic into the overlay network.
```
βββββββββββββββββββ DNS Query βββββββββββββββββββββ
β EKS Control β βββββββββββββΊ β CoreDNS β
β Plane β β (eksintercept) β
βββββββββββββββββββ βββββββββββ¬ββββββββββ
β
ββββββββββββββββββββββββββββ
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Is source IP from control plane? β
β AND is query for webhook service? β
ββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ
β
ββββββββββββββββ΄βββββββββββββββ
βΌ βΌ
ββββββββββββββββββββββ ββββββββββββββββββββββββββββ
β YES β β NO β
β Return Gateway β β Forward to kubernetes β
β Endpoint IP β β plugin β
ββββββββββββββββββββββ ββββββββββββββββββββββββββββ
```
The problem is that for now clusterDNS doesn't takes any part in EKS hostnames dns resolution.
**Are you currently working around this issue?**
Yes.
**Additional context**
1) This approach allows handling webhooks in EKS (with an overlay pod network) without any changes to webhook or Helm chart configuration.
2) no need for `hostNetwork`
3) No webhook pod ports collision
4) No need to manage many aws security group rules to open each webhook port.
At the moment, the EKS control-plane API server operates with `enable-aggregator-routing` enabled. This means that the control plane does not interact with the non-routable Kubernetes Service CIDR. Instead, the control plane queries Service endpoints and communicates with them directly.
This behavior can be changed.
When enable-aggregator-routing is set to false, the kube-apiserver uses the in-cluster DNS server to resolve Service names. The DNS server will behave similarly to aggregator routing: the API server will receive the IP address of one of the Ready service endpoints as dns response. However, this approach provides the ability to control "routing" at the DNS level.
This approach could be used for all services which a part of:
- MutatingWebhookConfiguration
- ValidatingWebhookConfiguration
- CRD conversion webhooks
- ApiService
/label proposal
/label eks
Contributor guide
Research direction
Start with the EKS control-plane DNS and enable-aggregator-routing behavior described in the issue, then review the linked Cilium reports and the referenced CoreDNS eksintercept plugin. Define how resolver configuration should apply to webhook and ApiService lookups, and verify that the requested behavior supports routing through a gateway without changing webhook configurations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, devops, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100