aws / aws/containers-roadmap

[EKS] [WINDOWS]: (Kube-Proxy/DNS) DNS randomly fails to resolve

Open
#2,176 18 comments 17 reactions 0 assignees View on GitHub
EKS EKS Networking Proposed Windows
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Kubernetes DNS fails randomly and doesn't recover.

We have an issue that transcends regions/accounts/clusters. Randomly a pod will start to exhibit dns lookup errors accessing k8s service addresses; but also in some cases accessing other services like rds clusters. The errors will look like standard dns resolution issues.

```
Exception msg System.Exception: Bad status code returned from xx service: 0 The remote name could not be resolved: 'xx-xx-internalapi-de-prod-svc.dealerplatform-de-prod.svc.cluster.local' System.Net.WebException: The remote name could not be resolved: 'xx-xx-internalapi-de-prod-svc.xx-de-prod.svc.cluster.local'
```

Executing an nslookup on a pod will also fail

```
C:\app>nslookup xx-xx-internalapi-de-prod-svc.xx-de-prod.svc.cluster.local
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 172.20.0.10

DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to UnKnown timed-out
```

Executing the same on a working node resolves fine.

The node in this case had been live since September and failed this morning at around 11:27am

This is possibly an issue for the windows-container repo (hcshim)/containerd; but as EKS is running multiple versions behind I'd prefer to have some idea where this sits. It 'could' be kube-proxy related as it maintains the service list but these issues only seem to occur in the eks amis so I'd prefer to prove it before spelunking into debugging remote services (again).
Or it could be related to the CNI which controls the bridge.

I've modified and ran amazon-eks-diag before replacing the node so I have the full dump.
[EKSDiag_20231013_130949.zip](https://github.com/aws/containers-roadmap/files/12896406/EKSDiag_20231013_130949.zip)

**Which service(s) is this request for?**
EKS

**Are you currently working around this issue?**

How are you currently solving this problem?
Killing nodes as they fail

**Additional context**
EKS 1.26
ami-0fe16592dc421d3f4 (amazon/Windows_Server-2019-English-Core-EKS_Optimized-1.26-2023.08.17)
eu-central-1 (eu-central-1b)
Windows_Server-2019-English-Core-EKS_Optimized-1.26-2023.08.17
Created 2023-09-04T12:16:30Z

c5.2xlarge
Kernel Version: 10.0.17763.4737

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Start by reviewing the attached EKSDiag dump and comparing a failing Windows node with a working node, then investigate the kube-proxy, DNS, and CNI paths described in the report. Done means identifying the component responsible for intermittent resolution failures and documenting or validating a fix beyond replacing nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.