Worker node is in unusable after powering off the VM from vSphere
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
After powering off the worker node from vSphere, it comes back up in an unusable state. It doesn't show an IP for the node
```
k get nodes -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
abhnvp-test-jl8nv Ready control-plane,master 3d4h v1.21.2-eks-1-21 198.18.153.36 198.18.153.36 Ubuntu 20.04.3 LTS 5.4.0-89-generic containerd://1.5.5
abhnvp-test-md-0-7f7d77778d-cht87 Ready 21m v1.21.2-eks-1-21 Ubuntu 20.04.3 LTS 5.4.0-90-generic containerd://1.5.5
abhnvp-test-md-0-7f7d77778d-dqvkp Ready 3d1h v1.21.2-eks-1-21 198.18.38.75 198.18.38.75 Ubuntu 20.04.3 LTS 5.4.0-89-generic containerd://1.5.5
```
It is able to schedule pods
```
k get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-eks-a-9644dd8dc-4qhtd 1/1 Running 0 64s 192.168.8.117 abhnvp-test-md-0-7f7d77778d-cht87
hello-eks-a-9644dd8dc-6x7xd 1/1 Running 0 63s 192.168.8.182 abhnvp-test-md-0-7f7d77778d-cht87
hello-eks-a-9644dd8dc-ghm7h 1/1 Running 0 63s 192.168.8.125 abhnvp-test-md-0-7f7d77778d-cht87
hello-eks-a-9644dd8dc-p587h 1/1 Running 0 63s 192.168.8.178 abhnvp-test-md-0-7f7d77778d-cht87
hello-eks-a-9644dd8dc-qgtxg 1/1 Running 0 63s 192.168.8.132 abhnvp-test-md-0-7f7d77778d-cht87
```
but the pods themselves aren't usable
```
k logs hello-eks-a-9644dd8dc-4qhtd
Error from server: Get "https://abhnvp-test-md-0-7f7d77778d-cht87:10250/containerLogs/default/hello-eks-a-9644dd8dc-4qhtd/hello": dial tcp: lookup abhnvp-test-md-0-7f7d77778d-cht87 on 10.2.192.12:53: no such host
```
When I tried with just CAPV, it all worked as expected and the node came back up with IPs showing and the pods were working as expected.
**What you expected to happen**:
Expected the worker node to power back up and work normally
**How to reproduce it (as minimally and precisely as possible)**:
- Create an EKS Anywhere cluster on vSphere using the CLI
- Manually power off one of the worker node VMs from the vSphere UI
- Wait for the worker node to power back up automatically
**Anything else we need to know?**:
**Environment**:
- OS: Ubuntu
- EKS Anywhere Release: v0.6.0
- EKS Distro Release: 1.21.2
Contributor guide
Research direction
Reproduce the vSphere workflow by powering off a worker VM and waiting for it to recover. Inspect `k get nodes -owide`, `k get pods -owide`, and `k logs` to compare the recovered node with the working node; done means the node reports its IP and pod logs are usable again.
Written by the indexing model from the issue text.
Assessment
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100