aws / aws/eks-anywhere

Worker node is in unusable after powering off the VM from vSphere

Open
#674 0 comments 0 reactions 0 assignees View on GitHub
area/providers/vsphere kind/bug team/cli
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
After powering off the worker node from vSphere, it comes back up in an unusable state. It doesn't show an IP for the node

```
k get nodes -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
abhnvp-test-jl8nv Ready control-plane,master 3d4h v1.21.2-eks-1-21 198.18.153.36 198.18.153.36 Ubuntu 20.04.3 LTS 5.4.0-89-generic containerd://1.5.5
abhnvp-test-md-0-7f7d77778d-cht87 Ready 21m v1.21.2-eks-1-21 Ubuntu 20.04.3 LTS 5.4.0-90-generic containerd://1.5.5
abhnvp-test-md-0-7f7d77778d-dqvkp Ready 3d1h v1.21.2-eks-1-21 198.18.38.75 198.18.38.75 Ubuntu 20.04.3 LTS 5.4.0-89-generic containerd://1.5.5
```

It is able to schedule pods
```
k get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-eks-a-9644dd8dc-4qhtd 1/1 Running 0 64s 192.168.8.117 abhnvp-test-md-0-7f7d77778d-cht87
hello-eks-a-9644dd8dc-6x7xd 1/1 Running 0 63s 192.168.8.182 abhnvp-test-md-0-7f7d77778d-cht87
hello-eks-a-9644dd8dc-ghm7h 1/1 Running 0 63s 192.168.8.125 abhnvp-test-md-0-7f7d77778d-cht87
hello-eks-a-9644dd8dc-p587h 1/1 Running 0 63s 192.168.8.178 abhnvp-test-md-0-7f7d77778d-cht87
hello-eks-a-9644dd8dc-qgtxg 1/1 Running 0 63s 192.168.8.132 abhnvp-test-md-0-7f7d77778d-cht87
```
but the pods themselves aren't usable
```
k logs hello-eks-a-9644dd8dc-4qhtd
Error from server: Get "https://abhnvp-test-md-0-7f7d77778d-cht87:10250/containerLogs/default/hello-eks-a-9644dd8dc-4qhtd/hello": dial tcp: lookup abhnvp-test-md-0-7f7d77778d-cht87 on 10.2.192.12:53: no such host
```

When I tried with just CAPV, it all worked as expected and the node came back up with IPs showing and the pods were working as expected.

**What you expected to happen**:
Expected the worker node to power back up and work normally

**How to reproduce it (as minimally and precisely as possible)**:
- Create an EKS Anywhere cluster on vSphere using the CLI
- Manually power off one of the worker node VMs from the vSphere UI
- Wait for the worker node to power back up automatically

**Anything else we need to know?**:

**Environment**:
- OS: Ubuntu
- EKS Anywhere Release: v0.6.0
- EKS Distro Release: 1.21.2

Contributor guide

Open the contributing guide

Research direction

Reproduce the vSphere workflow by powering off a worker VM and waiting for it to recover. Inspect `k get nodes -owide`, `k get pods -owide`, and `k logs` to compare the recovered node with the working node; done means the node reports its IP and pod logs are usable again.

Written by the indexing model from the issue text.

Assessment

Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.