cluster failed to recover post deleting all master machines at once
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
**What you expected to happen**:
- When new machines get created, one of machine should automatically get floating ip and replace stale old master ip addresses in ETCD and API Server manifests
**How to reproduce it (as minimally and precisely as possible)**:
- Delete all master machines and validate not able to connect to the cluster
**Anything else we need to know?**:
Workaround steps,
1. The new restored machines had new IPs but the state of Master Component was based on old IPs.
2. We assigned floating IP ([controlPlaneConfiguration.endpoint.host ](http://controlplaneconfiguration.endpoint.host/)) [1] manually to one of the master machine.
3. Modify ETCD and API server manifests and replace old master IP addresses with new IPs
4. Fix the DNS resolution in Master Machines
5. Recycle APIServer, ETCD and kubelet from Master Machines
**Environment**:
- EKS Anywhere Release:
- EKS Distro Release:
Contributor guide
Research direction
Start by reproducing the failure described after deleting all master machines and compare the restored machines' IPs with the stale addresses in the etcd and API server manifests. Review controlPlaneConfiguration.endpoint.host, DNS resolution on the master machines, and the listed APIServer, etcd, and kubelet recovery steps. Done means the cluster recovers automatically with a new machine receiving the floating IP and updated control-plane addresses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100