aws / aws/containers-roadmap

[EKS] [Windows] Invalid Hostname override passed to kube-proxy

Open
#273 6 comments 0 reactions 0 assignees View on GitHub
EKS Windows
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

There seems to be an issue with kube-proxy on windows containers receiving the wrong Hostname Override parameter from the bootstrap script. Looking at kube-proxy logs, it fails to lookup the ip (from the kubernetes api) for the node and falls back to 127.0.0.1, causing issues.

From there there are errors creating the proxy endpoints. The nodes do appear to work, until you create a deployment with a service and more then 1 replica. With a single replica, everything works, but adding a second (or more) all pods fail.

I can see what causes the error, and have a patch for my specific cluster configuration, but I am not sure why it is necessary for this cluster vs others not having the issues. One guess - this is an older cluster that has been upgraded from 1.10 all the way to current 1.11

---
`Start-EKSBootstrap.ps1` has a `$HostName` variable from calling the metadata endpoint. The value of that, for an example node, is `ip-10-20-66-170`. That value is then passed as an arg to kubelet and kube-proxy. The issue comes from the nodes actual name ` ip-10-20-66-170.ec2.internal `.

That value is then used by kube-proxy to lookup the IP [here](https://github.com/kubernetes/kubernetes/blob/7a578febe155a7366767abce40d8a16795a96371/cmd/kube-proxy/app/server.go#L6220). Using `ip-10-20-66-170` returns nil, because that lookup is by name. Using ` ip-10-20-66-170.ec2.internal ` works as expected, returning the correct id.

As a quick patch, I added this script block to patch the Bootstrap script to use the "correct" hostname for kube-proxy

```ps1
function patchBootstrap() {
$target = "[string]`$EniMACAddress = Get-EC2MetaData 'latest/meta-data/mac'"
$newStanza = "`$HostName = `$HostName + '.ec2.internal'"
$replacement = "$target `n $newStanza"

$script = [System.IO.File]::ReadAllText("C:\Program Files\Amazon\EKS\Start-EKSBootstrap.ps1").Replace($target, $replacement)
$script | Set-Content -Path "C:\Program Files\Amazon\EKS\Start-EKSBootstrap.ps1" -Force
}
```

I didn't see the repo for the AMI, so I wasn't sure if there was more documentation for the Bootstrap that would help explain some of this.

---
AMI - `ami-032bdf5292844295a`
EKS - `1.11` - `eks.2` - Updated from 1.10 to 1.11
Kubelet - ` v1.11.5`
Kube-Proxy - ` v1.11.5`
aws-node - `amazon-k8s-cni:v1.3.3`

Example windows node:

```
Name: ip-10-20-66-170.ec2.internal
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=m3.large
beta.kubernetes.io/os=windows
failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1d
kubernetes.io/hostname=ip-10-20-66-170
mhelpdesk.com/ocean-node=true
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp: Wed, 01 May 2019 11:33:15 -0400
Taints: os=Win1809:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
OutOfDisk False Wed, 01 May 2019 13:37:26 -0400 Wed, 01 May 2019 11:33:14 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Wed, 01 May 2019 13:37:26 -0400 Wed, 01 May 2019 11:33:14 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 01 May 2019 13:37:26 -0400 Wed, 01 May 2019 11:33:14 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 01 May 2019 13:37:26 -0400 Wed, 01 May 2019 11:33:14 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 01 May 2019 13:37:26 -0400 Wed, 01 May 2019 11:33:14 -0400 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.20.66.170
InternalDNS: ip-10-20-66-170
Hostname: ip-10-20-66-170
Capacity:
cpu: 2
ephemeral-storage: 0
memory: 7863920Ki
pods: 110
vpc.amazonaws.com/ENI: 1
vpc.amazonaws.com/PrivateIPv4Address: 9
Allocatable:
cpu: 2
ephemeral-storage: 0
memory: 7863920Ki
pods: 110
vpc.amazonaws.com/ENI: 1
vpc.amazonaws.com/PrivateIPv4Address: 9
System Info:
Machine ID: QA-EKSW-07e4f48
System UUID:
Boot ID:
Kernel Version: 10.0.17763.316

OS Image: Windows Server 2019 Datacenter
Operating System: windows
Architecture: amd64
Container Runtime Version: docker://18.9.2
Kubelet Version: v1.11.5
Kube-Proxy Version: v1.11.5
ExternalID: ip-10-20-66-170.ec2.internal
ProviderID: aws:///us-east-1d/i-07e4f48c4572a74dd
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
qa mapi-55f5bf5886-zl7qz 0 (0%) 0 (0%) 0 (0%) 0 (0%)
qa mhdv2-8645bd495-ngch4 0 (0%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
0 (0%) 0 (0%) 0 (0%) 0 (0%)
Events:

```

Contributor guide

Open the contributing guide

Research direction

Start with Start-EKSBootstrap.ps1 and the kube-proxy node IP lookup in server.go referenced by the issue. Compare the hostname passed to kubelet and kube-proxy with the node's ExternalID and observed logs, then verify that the corrected override resolves the node IP and allows services with multiple replicas to create proxy endpoints.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, powershell
Domain
devops, infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.