microsoft / microsoft/hcsshim

HNS loadbalancer backend pods malfunction

Open
#994 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
694
Forks
304
Avg merge
1d 19h
Merged PRs (30d)
28

Description

Hello,

I deployed a kuberenetes cluster with windows workers on openstack infrastructure.

What happen:

  • We lost pods dns resolution on worker nodes.
  • Pod can't reach de core dns service.
  • We facing Windows Host Network Service (HNS) malfunction which cause loss of pods dns resolution on the windows server.
  • It seems to be a malfunction between HNS loadbalancer and HNS pod backends.
  • The issue disappears after restarting the windows Server.
  • pod dns resolution work fine if i reach it with coredns pod ip address
  • kubectl exec -it win-pod -- powershell.exe -Command resolve-dnsname kube-dns.kube-system.svc.cluster.local
    KO with timeout error : resolve-dnsname : kube-dns.kube-system.svc.cluster.local : This operation returned because the timeout period expired
  • kubectl exec -it win-pod -- powershell.exe -Command resolve-dnsname kube-dns.kube-system.svc.cluster.local -Server 10.237.25.1
    OK (10.237.25.1 is coredns pod ip address)
  • i followed this doc: https://argonsys.com/microsoft-cloud/library/troubleshooting-kubernetes-networking-on-windows-part-1/ and execute https://github.com/microsoft/SDN/blob/master/Kubernetes/windows/debug/collectlogs.ps1 (everything seems to be ok)

To reproduce the issue:

  • Deploy/delete 5 pods on windows node each one minutes 20 times

Environment:
• Control plane (k8s Master Nodes):
o Linux CentOS-7.5
o Kuberenetes 1.18.3
o Docker 1.13.1
o CNI plugin Calico version: 3.17

• Windows server (k8s Worker Nodes):
o Windows Server 2019 Standard
o Version: 1809
o OS build: 17763.737
o Docker EE version: 19.3.13
o Networking : l2bridge with non-DSR mode

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported reproduction: deploy and delete five pods on a Windows node repeatedly, then compare resolve-dnsname through kube-dns with resolution through the CoreDNS pod IP. Review the referenced collectlogs.ps1 output and the HNS load balancer/backend state. Done means the HNS failure is reproducible and its cause or a verified fix is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes
Domain
networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.