aws / aws/eks-anywhere

MHC doesn't work until a cluster is fully formed

Open
#3,979 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
CAPI's MHC is currently only useful in a fully formed cluster. It's currently unable to detect or remediate issues in an EKS-A cluster if those issues happen while the cluster is being created. Possible issues could include loss of network connectivity to a machine, hardware failure, or misconfigured machines that fail to boot.

We identified three scenarios that each need their own fixes:
1. A worker machine fails during cluster creation. With some simple changes in EKS-A, CAPI will handle this situation.
- In cluster_manager.go, CreateWorkloadCluster() waits for all CP nodes to be ready before it returns. Instead, it should only wait for the first CP node (the cluster endpoint) to be ready.
- In create.go, Run() should install networking on the cluster as soon as the CP endpoint is ready and CreateWorkloadCluster() returns. This will allow MHC to start getting node information from the new cluster.
- In create.go, Run() should add the machine health checks right after the networking is installed.
2. The first CP machine or an unstacked etcd machine fails during cluster creation. This leaves the cluster without an API endpoint, so it's useless. We discussed this in a CAPI meeting, and the recommendation we received was to delete the cluster and start over. This seems reasonable. If the cluster endpoint isn't available in a reasonable amount of time, EKS-A should delete the cluster and try again.
3. A subsequent CP machine (not the first one) fails during cluster creation. This should work a lot like a failed worker machine, but there's an additional complication. CAPI won't remediate any machines until all CP machines have joined the cluster. As a result, if one CP machine fails to join the cluster, nothing can be remediated. There's an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues/7496) to address this. Once that's fixed in CAPI, it's posible that the same EKS-A changes that work for worker machines will also work for these CP machines.

**What you expected to happen**:
EKS-A and CAPI should be resilient enough to to detect and remediate these failures. Instead, EKS-A simply waits until it times out, leaving the cluster in a partially created state.

**How to reproduce it (as minimally and precisely as possible)**:
Create a workload cluster, ensuring that one of the machines is isolated from the others. This can be accomplished by placing the machines in different networks that aren't connected to each other. Another method is to connect to one of the machines and disable its network interfaces as soon as it boots, before it can join the cluster.

**Anything else we need to know?**:

**Environment**:
- EKS Anywhere Release: 0.12.0
- EKS Distro Release: 0.115.0

Contributor guide

Open the contributing guide

Research direction

Start with cluster_manager.go and create.go, tracing CreateWorkloadCluster() and Run() through workload-cluster creation, networking installation, and machine health check setup. Compare the three failure scenarios with CAPI issue #7496; done means cluster creation can detect and remediate the described failures instead of timing out in a partially created state.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.