aws / aws/eks-anywhere

CloudStack clusterctl move occasionally fails to bring over cloudstackmachinetemplate for machinedeployments

Open
#2,183 5 comments 0 reactions 0 assignees View on GitHub
area/providers/capc kind/bug team/providers
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
In the e2e tests, we run clusterctl move soon after a cluster is first created. It seems like the cloudstackmachinetemplates occasionally do not get inherited by the CAPI cluster in time so the move does not bring them over to the destination cluster and so the move ultimately fails with error messages like

```
2022-05-16T16:53:25.110-0400 V3 Waiting for workload cluster machine deployment replicas to be ready after move
2022-05-16T16:53:25.110-0400 V6 waiting for nodes {"cluster": "eksa-drib-a9ee116"}
2022-05-16T16:53:25.110-0400 V6 Executing command {"cmd": "/usr/local/bin/docker exec -i eksa_1652734226607009000 kubectl get machinedeployments.cluster.x-k8s.io -o json --kubeconfig eksa-drib-a9ee116/generated/eksa-drib-a9ee116.kind.kubeconfig --namespace eksa-system"}
2022-05-16T16:53:25.777-0400 V6 waiting for nodes {"cluster": "eksa-drib-a9ee116"}
2022-05-16T16:53:25.777-0400 V6 Executing command {"cmd": "/usr/local/bin/docker exec -i eksa_1652734226607009000 kubectl get machinedeployments.cluster.x-k8s.io -o json --kubeconfig eksa-drib-a9ee116/generated/eksa-drib-a9ee116.kind.kubeconfig --namespace eksa-system"}
2022-05-16T16:53:26.304-0400 V5 Error happened during retry {"error": "machine deployment is in phase", "retries": 1}
2022-05-16T16:53:26.304-0400 V5 Sleeping before next retry {"time": "0s"}
...
2022-05-16T15:21:11.540-0400 V6 Executing command {"cmd": "/usr/local/bin/docker exec -i eksa_1652726909294702000 kubectl get machinedeployments.cluster.x-k8s.io -o json --kubeconfig eksa-drib-2ffeb29/generated/eksa-drib-2ffeb29.kind.kubeconfig --namespace eksa-system"}
2022-05-16T15:21:12.480-0400 V5 Error happened during retry {"error": "machine deployment is in phase", "retries": 2292}
2022-05-16T15:21:12.481-0400 V5 Sleeping before next retry {"time": "0s"}
2022-05-16T15:21:12.481-0400 V5 Timeout reached. Returning error {"retries": 2292, "duration": "30m0.809020736s", "error": "machine deployment is in phase"}
2022-05-16T15:21:12.482-0400 V4 Task finished {"task_name": "cluster-management-move", "duration": "30m37.551740161s"}
...
Error: failed to delete cluster: waiting for workload cluster machinedeployment replicas to be ready: retries exhausted waiting for machinedeployment replicas to be ready: machine deployment is in phase
```

**What you expected to happen**:
I expected the move to succeed and bring the cloudstackmachinetemplate for the machinedeployment over

**How to reproduce it (as minimally and precisely as possible)**:
This is a nondeterministic bug. It appears on any e2e test where we first create a cluster, and then proceed to either upgrade or delete it.

**Anything else we need to know?**:
I discovered a workaround is to manually move the cloudstackmachinetemplate, and then force the associated capi machinedeployment to reconcile by editing some field in its spec

**Environment**:
- EKS Anywhere Release: 0.9.0
- EKS Distro Release:

Contributor guide

Open the contributing guide

Research direction

Start with the e2e tests that create a cluster and then run clusterctl move, and inspect how CloudStackMachineTemplate objects are handled for MachineDeployments. Reproduce the nondeterministic failure and compare the destination cluster before and after the manual template move and MachineDeployment reconciliation. Done means clusterctl move consistently transfers the template and the move completes successfully without the workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.