aws / aws/containers-roadmap

[EKS] [request]: Nodelocal DNS Cache

Open
#303 47 comments 238 reactions 0 assignees View on GitHub
EKS EKS Networking Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

**Tell us about your request**
I would like an officially documented and supported method for installing the Kubernetes Node Local DNS Cache Addon.

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Kubernetes clusters with a high request rate often experience high rates of failed DNS lookups. For example this affects us when using the AWS SDKs particularly with Alpine / musl-libc containers.

The _Nodelocal DNS Cache_ aims to resolve this (together with kernel patches in 5.1 to fix a conntrack race condition).

[Nodelocal DNS Addon](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns/nodelocaldns)

Kubeadm is aiming to support Nodelocal-dns-cache in 1.15. [k/k #70707](https://github.com/kubernetes/kubernetes/issues/70707)

**Are you currently working around this issue?**
Retrying requests at the application level which fail due to DNS errors.

**Additional context**
Kubernetes DNS Issues include:
- Linux Kernel bug in netfilter conntrack (fixed in kernel 5.1) [1][2]
- Exacerbated by musl-libc behaviour of issuing parallel queries, which is widely used in Alpine Docker containers. musl-libc does not respect the resolv.conf `single-request` option and it appears this will not be changed [2][3]
- The AWS EC2 limit of 1024 Packets Per Second Per ENI [5]

**Attachments**
[0] https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#retryDelayOptions-property
[1] https://lkml.org/lkml/2019/2/28/707
[2] https://blog.quentin-machu.fr/2018/06/24/5-15s-dns-lookups-on-kubernetes/
[3] https://www.weave.works/blog/racy-conntrack-and-dns-lookup-timeouts
[4] https://www.openwall.com/lists/musl/2015/10/22/15
[5] https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-limits

Contributor guide

Open the contributing guide

Research direction

Read the linked Kubernetes NodeLocal DNS Addon documentation and Kubernetes issue #70707 first, then compare them with the EKS request. Done means EKS has an officially documented and supported installation method for NodeLocal DNS Cache, including guidance relevant to the DNS lookup failures described here.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, linux
Domain
cloud, infrastructure, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.