aws / aws/eks-anywhere

cluster failed to recover post deleting all master machines at once

Open
#6,152 0 comments 0 reactions 0 assignees View on GitHub
external kind/bug
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:

**What you expected to happen**:
- When new machines get created, one of machine should automatically get floating ip and replace stale old master ip addresses in ETCD and API Server manifests

**How to reproduce it (as minimally and precisely as possible)**:
- Delete all master machines and validate not able to connect to the cluster

**Anything else we need to know?**:

Workaround steps,

1. The new restored machines had new IPs but the state of Master Component was based on old IPs.
2. We assigned floating IP ([controlPlaneConfiguration.endpoint.host ](http://controlplaneconfiguration.endpoint.host/)) [1] manually to one of the master machine.
3. Modify ETCD and API server manifests and replace old master IP addresses with new IPs
4. Fix the DNS resolution in Master Machines
5. Recycle APIServer, ETCD and kubelet from Master Machines

**Environment**:
- EKS Anywhere Release:
- EKS Distro Release:

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure described after deleting all master machines and compare the restored machines' IPs with the stale addresses in the etcd and API server manifests. Review controlPlaneConfiguration.endpoint.host, DNS resolution on the master machines, and the listed APIServer, etcd, and kubelet recovery steps. Done means the cluster recovers automatically with a new machine receiving the floating IP and updated control-plane addresses.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.