aws / aws/containers-roadmap

[EKS] [request]: ability to set dns resolving for EKS.

Open
#2,744 0 comments 4 reactions 0 assignees View on GitHub
EKS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a πŸ‘ [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
It'll be good to set dns resolver for EKS ControlPlane.
It could be any DNS server reachable in VPC (e.g. CoreDNS running on hostNetwork on workload)

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
There is well know issue with webhooks in EKS for non VPC CNI, e.g
https://github.com/cilium/cilium/issues/29454
https://github.com/cilium/cilium/issues/30111
https://github.com/cilium/cilium/issues/21959

EKS Control-plane unable to call webhooks running in overlay network. There is no communication between EKS CP and cilium PodCIDR.
I have idea to implement approach to fix this.

If `enable-aggregator-routing` set to false on api-server, all webhook service reference could be resolved by internal names like:
http://..kubernetes.svc.cluster.local

I want to intercept this dns query on the side of coreDNS (with plugin) running in k8s:
https://github.com/CharlieR-o-o-t/coredns_eks/tree/master/plugin/eksintercept

Then, using DNS, the traffic is routed to a dedicated webhook gateway service endpoint whose pods run with hostNetwork enabled.
The gateway listens on all webhook ports, performs SSL passthrough, supports multiple webhook configurations on the same port, and forwards L3 traffic into the overlay network.

```
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” DNS Query β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ EKS Control β”‚ ────────────► β”‚ CoreDNS β”‚
β”‚ Plane β”‚ β”‚ (eksintercept) β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Is source IP from control plane? β”‚
β”‚ AND is query for webhook service? β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β–Ό β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ YES β”‚ β”‚ NO β”‚
β”‚ Return Gateway β”‚ β”‚ Forward to kubernetes β”‚
β”‚ Endpoint IP β”‚ β”‚ plugin β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

```

The problem is that for now clusterDNS doesn't takes any part in EKS hostnames dns resolution.

**Are you currently working around this issue?**
Yes.

**Additional context**
1) This approach allows handling webhooks in EKS (with an overlay pod network) without any changes to webhook or Helm chart configuration.
2) no need for `hostNetwork`
3) No webhook pod ports collision
4) No need to manage many aws security group rules to open each webhook port.

At the moment, the EKS control-plane API server operates with `enable-aggregator-routing` enabled. This means that the control plane does not interact with the non-routable Kubernetes Service CIDR. Instead, the control plane queries Service endpoints and communicates with them directly.

This behavior can be changed.

When enable-aggregator-routing is set to false, the kube-apiserver uses the in-cluster DNS server to resolve Service names. The DNS server will behave similarly to aggregator routing: the API server will receive the IP address of one of the Ready service endpoints as dns response. However, this approach provides the ability to control "routing" at the DNS level.

This approach could be used for all services which a part of:

- MutatingWebhookConfiguration

- ValidatingWebhookConfiguration

- CRD conversion webhooks

- ApiService

/label proposal
/label eks

Contributor guide

Open the contributing guide

Research direction

Start with the EKS control-plane DNS and enable-aggregator-routing behavior described in the issue, then review the linked Cilium reports and the referenced CoreDNS eksintercept plugin. Define how resolver configuration should apply to webhook and ApiService lookups, and verify that the requested behavior supports routing through a gateway without changing webhook configurations.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, devops, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.