[EKS] [WINDOWS]: (Kube-Proxy/DNS) DNS randomly fails to resolve
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Kubernetes DNS fails randomly and doesn't recover.
We have an issue that transcends regions/accounts/clusters. Randomly a pod will start to exhibit dns lookup errors accessing k8s service addresses; but also in some cases accessing other services like rds clusters. The errors will look like standard dns resolution issues.
```
Exception msg System.Exception: Bad status code returned from xx service: 0 The remote name could not be resolved: 'xx-xx-internalapi-de-prod-svc.dealerplatform-de-prod.svc.cluster.local' System.Net.WebException: The remote name could not be resolved: 'xx-xx-internalapi-de-prod-svc.xx-de-prod.svc.cluster.local'
```
Executing an nslookup on a pod will also fail
```
C:\app>nslookup xx-xx-internalapi-de-prod-svc.xx-de-prod.svc.cluster.local
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 172.20.0.10
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to UnKnown timed-out
```
Executing the same on a working node resolves fine.
The node in this case had been live since September and failed this morning at around 11:27am
This is possibly an issue for the windows-container repo (hcshim)/containerd; but as EKS is running multiple versions behind I'd prefer to have some idea where this sits. It 'could' be kube-proxy related as it maintains the service list but these issues only seem to occur in the eks amis so I'd prefer to prove it before spelunking into debugging remote services (again).
Or it could be related to the CNI which controls the bridge.
I've modified and ran amazon-eks-diag before replacing the node so I have the full dump.
[EKSDiag_20231013_130949.zip](https://github.com/aws/containers-roadmap/files/12896406/EKSDiag_20231013_130949.zip)
**Which service(s) is this request for?**
EKS
**Are you currently working around this issue?**
How are you currently solving this problem?
Killing nodes as they fail
**Additional context**
EKS 1.26
ami-0fe16592dc421d3f4 (amazon/Windows_Server-2019-English-Core-EKS_Optimized-1.26-2023.08.17)
eu-central-1 (eu-central-1b)
Windows_Server-2019-English-Core-EKS_Optimized-1.26-2023.08.17
Created 2023-09-04T12:16:30Z
c5.2xlarge
Kernel Version: 10.0.17763.4737
Contributor guide
Research direction
No source file or test is named. Start by reviewing the attached EKSDiag dump and comparing a failing Windows node with a working node, then investigate the kube-proxy, DNS, and CNI paths described in the report. Done means identifying the component responsible for intermittent resolution failures and documenting or validating a fix beyond replacing nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- networking, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100