aws / aws/eks-anywhere

unhealthyMachineTimeout not working when VM is powered off (VM not deleted from disk)

Open
#8,785 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:

I have created EKSA Cluster with following configuration,

1. unhealthyMachineTimeout set to 30 seconds (minimum value) in the Cluster config file Worker node section
2. Enabled Autoscaling configuration in cluster config file for worker nodes
3. Installed Cluster Autoscaler curated package on the cluster

I went through two scenarios post cluster creation,

1. **Scenario 1:** Navigate to VMWare vSphere console, Click on one of worker node, Right Click and Power Off
2. **Scenario 2:** Click on one of worker node, Right Click > Power Off, Right click again > Delete from the disk

Scenario 1 fails all the time. No new node is created. capv pod logs do not show any event that node is unhealthy until 4-5 minutes. And then, node either gets deleted and new node is provisioned or node gets powered on.

Scenario 2 works all the time. Post deletion of node, new node gets provisioned within 30 seconds.

[1] https://anywhere.eks.amazonaws.com/docs/getting-started/optional/healthchecks/#__machinehealthcheckunhealthymachinetimeout__-optional

**What you expected to happen**:

For scenario 1, capv should respect unhealthyMachineTimeout 30 seconds value. When unhealthyMachineTimeout is set to 5 minutes, capv takes around 20-40 minutes to realize the node is powered off or not ready.

I am not sure if we need something like a node termination handler that Amazon EKS on cloud has.

**How to reproduce it (as minimally and precisely as possible)**:
- Configure worker node section of Cluster config file as following.

```
workerNodeGroupConfigurations:
- count: 1
machineGroupRef:
kind: VSphereMachineConfig
name: demo-mgmt
name: md-0
autoscalingConfiguration:
minCount: 1
maxCount: 5
machineHealthCheck:
unhealthyMachineTimeout: 30s
maxUnhealthy: 100%
```

**Anything else we need to know?**:

**Environment**: EKSA with vSphere
- EKS Anywhere Release: 0.20

```
Version: v0.20.4
Release Manifest URL: https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml Bundle Manifest URL: https://anywhere-assets.eks.amazonaws.com/releases/bundles/74/manifest.yaml
```

- EKS Distro Release: not sure

Contributor guide

Open the contributing guide

Research direction

Start with the workerNodeGroupConfigurations.machineHealthCheck example and the linked MachineHealthCheck documentation, then compare capv pod behavior for a powered-off VM versus one deleted from disk. Reproduce the issue with unhealthyMachineTimeout set to 30s in EKS Anywhere 0.20.4; done means a powered-off worker is detected and replaced within the configured timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.