aws / aws/eks-anywhere

Starting EKS Anywhere on VMware, etcd fails to start

Open
#9,047 26 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:

tmcooper@ubuntu-server-2404:~$ eksctl anywhere create cluster -f eksa-w01-cluster.yaml
Warning: VSphereDatacenterConfig configured in insecure mode
Performing setup and validations
Warning: VSphereDatacenterConfig configured in insecure mode
✅ Connected to server
✅ Authenticated to vSphere
✅ Datacenter validated
✅ Network validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Machine config tags validated
✅ Control plane and Workload templates validated
✅ administrator@vsphere.local user vSphere privileges validated
✅ Vsphere Provider setup is valid
✅ Validate OS is compatible with registry mirror configuration
✅ Validate certificate for registry mirror
✅ Validate authentication for git provider
✅ Validate cluster's eksaVersion matches EKS-A version
✅ Validate cluster's kubelet configuration for Bottlerocket OS
✅ Validate cluster's worker node kubelet configuration for Bottlerocket OS
Creating new bootstrap cluster
Provider specific pre-capi-install-setup on bootstrap cluster
Installing cluster-api providers on bootstrap cluster
Provider specific post-setup
Installing EKS-A custom components on bootstrap cluster
Installing EKS-D components
Installing EKS-A custom components (CRD and controller)
Creating new management cluster
(gets hung)

From the log where it is repeating the problem:
{"T":1733494047021445202,"M":"Sleeping before next retry","time":"1s"}
{"T":1733494048021896267,"M":"Executing command","cmd":"/usr/bin/docker exec -i eksa_1733493551451751668 kubectl get --ignore-not-found -o json --kubeconfig kn01/generated/kn01.kind.kubeconfig Cluster.v1alpha1.anywhere.eks.amazonaws.com --namespace default kn01"}
{"T":1733494048518935993,"M":"Cluster generation and observedGeneration","Generation":1,"ObservedGeneration":1}
{"T":1733494048519028611,"M":"Error happened during retry","error":"cluster condition ControlPlaneReady is False: Etcd is not ready","retries":59}
{"T":1733494048519061258,"M":"Sleeping before next retry","time":"1s"}

_Why is etcd not becoming ready and does it have some log?_

The file with the YAML configuration is attached.

**What you expected to happen**:

My EKS Kubernetes to be set up

**How to reproduce it (as minimally and precisely as possible)**:

Set up eks anywhere, docker,
set environment passwords for VMWare
export EKSA_VSPHERE_USERNAME=administrator@vsphere.local
export EKSA_VSPHERE_PASSWORD=
run
eksctl anywhere create cluster -f eksa-w01-cluster.yaml

**Anything else we need to know?**:

**Environment**:
- Ubuntu 24.04.1 LTS (GNU/Linux 6.8.0-49-generic x86_64)
- eksctl 0.197.0
- VMWare:
Client version: 2.14.0
Client build number: 21993070
ESXi version: 8.0.2
ESXi build number: 22380479
- Vcenter 8
- EKS Anywhere Release: 0.197.0
- EKS Distro Release:
- kubectl version
Client Version: v1.31.3
Kustomize Version: v5.4.2

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with eksctl anywhere create cluster -f eksa-w01-cluster.yaml and inspect the retry output for the ControlPlaneReady and Etcd conditions. Find the management-cluster and etcd logs associated with the repeated kubectl get command; done means the management cluster completes creation with etcd becoming ready.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.