aws / aws/eks-anywhere

Cluster creation failing with

Open
#6,251 2 comments 0 reactions 0 assignees View on GitHub
area/cli external
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
Tried to spin up new cluster(30 nodes cluster) with the with management cluster in vsphere, but it failed with below error message
`Error: waiting for control plane to be ready: executing wait: executing wait: error: timed out waiting for the condition on clusters/<>`. When I checked the machines status in mgmt cluster(kg machines -n eksa-system) it showed all etcd nodes in the running state, control plane nodes in the provisioning state and worker nodes in the Pending state.

- There after when I tried to create the cluster again, it says
- `Validation failed {"validation": "validate cluster name", "error": "cluster name cluster_name already exists", "remediation": ""}`
- `Validation failed {"validation": "validate management cluster bundle version compatibility", "error": "cannot upgrade workload cluster with bundle spec.number 41 while management cluster mgmt is on older bundle spec.number 40", "remediation": "upgrade management cluster eksa-mgmt before creating workload cluster cluster_name"}`
- But when I tried to upgrade mgmt cluster, it is getting failed with below error message
- `Error: failed to upgrade cluster: backing up CAPI resources of management cluster before moving to bootstrap cluster: failed taking backup of CAPI objects: Moving to directory...
Discovering Cluster API objects
Error: failed to get object graph: failed to check for provisioned infrastructure: [cannot start the move operation while the control plane for "/, Kind=" eksa-system/worker_cluster is not yet initialized, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-bv557 is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-hvwrf is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-f4h5w is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-htv5p is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-rwxxq is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-d5jzt is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-mtrgl is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-dg4g9 is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-5zv2w is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-vktrw is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-mb6zp is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-2xmn5 is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-kzzz5 is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-v7prd is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-fb5l9 is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-ttp86 is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-mnccj is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-gwx4h is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-fh5b5 is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-hsnpp is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-lqbgl is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-khsht is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-qlh9c is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-hhvtg is still provisioning the node, cannot start the move operation while "/, Kind=" eksa-system/worker_cluster-wn-xxxxxxxxxxxxxxxxxxxxx-42jhz is still provisioning the node]`

We are struck in middle, everytime I'm seeing the new issues when I tried to spin up the new cluster, by which we are in a dilema to use whether EKSA or not. Most of the time we are seeing cluster api related issues

**What you expected to happen**:
It should bring up the new cluster without any issues or clean/upgrade the exisitng cluster

**How to reproduce it (as minimally and precisely as possible)**:
- Spin up mgmt cluster
- Use mgmt cluster to created worker cluster wait it to fail to replicate the issue

**Anything else we need to know?**:

**Environment**: Prod
- EKS Anywhere Release: v0.16.2
- EKS Distro Release:

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the v0.16.2 failure with a vSphere management cluster and a 30-node workload cluster, then inspect the reported control-plane timeout and machine states with `kg machines -n eksa-system`. Trace the subsequent cluster-name, bundle-version, and management-cluster upgrade errors. Done would require reliable cluster creation and a clean recovery or upgrade path after the initial failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.