HNS loadbalancer backend pods malfunction
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 694
- Forks
- 304
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 28
Description
Hello,
I deployed a kuberenetes cluster with windows workers on openstack infrastructure.
What happen:
- We lost pods dns resolution on worker nodes.
- Pod can't reach de core dns service.
- We facing Windows Host Network Service (HNS) malfunction which cause loss of pods dns resolution on the windows server.
- It seems to be a malfunction between HNS loadbalancer and HNS pod backends.
- The issue disappears after restarting the windows Server.
- pod dns resolution work fine if i reach it with coredns pod ip address
- kubectl exec -it win-pod -- powershell.exe -Command resolve-dnsname kube-dns.kube-system.svc.cluster.local
KO with timeout error : resolve-dnsname : kube-dns.kube-system.svc.cluster.local : This operation returned because the timeout period expired - kubectl exec -it win-pod -- powershell.exe -Command resolve-dnsname kube-dns.kube-system.svc.cluster.local -Server 10.237.25.1
OK (10.237.25.1 is coredns pod ip address) - i followed this doc: https://argonsys.com/microsoft-cloud/library/troubleshooting-kubernetes-networking-on-windows-part-1/ and execute https://github.com/microsoft/SDN/blob/master/Kubernetes/windows/debug/collectlogs.ps1 (everything seems to be ok)
To reproduce the issue:
- Deploy/delete 5 pods on windows node each one minutes 20 times
Environment:
• Control plane (k8s Master Nodes):
o Linux CentOS-7.5
o Kuberenetes 1.18.3
o Docker 1.13.1
o CNI plugin Calico version: 3.17
• Windows server (k8s Worker Nodes):
o Windows Server 2019 Standard
o Version: 1809
o OS build: 17763.737
o Docker EE version: 19.3.13
o Networking : l2bridge with non-DSR mode
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported reproduction: deploy and delete five pods on a Windows node repeatedly, then compare resolve-dnsname through kube-dns with resolution through the CoreDNS pod IP. Review the referenced collectlogs.ps1 output and the HNS load balancer/backend state. Done means the HNS failure is reproducible and its cause or a verified fix is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes
- Domain
- networking, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100