kind cluster deletion failed after upgrade
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
Attempted upgrade of management cluster managing 100 workload clusters. After the upgrade finished, the CLI attempted to delete the kind cluster and I saw
```
2022-11-07T16:51:35.082-0500 V4 Task start {"task_name": "delete-kind-cluster"}
2022-11-07T16:51:35.083-0500 V6 Executing command {"cmd": "/usr/local/bin/docker exec -i eksa_1667850170484856000 kind get clusters"}
2022-11-07T16:51:37.088-0500 V5 Executed kind get clusters {"response": "baseline-test-eks-a-cluster\n"}
2022-11-07T16:51:37.102-0500 V6 Executing command {"cmd": "/usr/local/bin/docker exec -i eksa_1667850170484856000 kubectl get crd clusters.cluster.x-k8s.io --kubeconfig baseline-test/generated/baseline-test.kind.kubeconfig"}
2022-11-07T16:51:40.678-0500 V6 Executing command {"cmd": "/usr/local/bin/docker exec -i eksa_1667850170484856000 kubectl get clusters.cluster.x-k8s.io -o json --kubeconfig baseline-test/generated/baseline-test.kind.kubeconfig --namespace eksa-system"}
2022-11-07T16:51:43.015-0500 V4 Deleting kind cluster {"name": "baseline-test-eks-a-cluster"}
2022-11-07T16:51:43.016-0500 V6 Executing command {"cmd": "/usr/local/bin/docker exec -i eksa_1667850170484856000 kind delete cluster --name baseline-test-eks-a-cluster"}
2022-11-07T16:52:00.397-0500 V9 docker {"stderr": "Deleting cluster \"baseline-test-eks-a-cluster\" ...\nERROR: failed to delete cluster \"baseline-test-eks-a-cluster\": failed to delete nodes: command \"docker rm -f -v baseline-test-eks-a-cluster-control-plane\" failed with error: exit status 1\n\nCommand Output: Error response from daemon: Could not kill running container 43583218989e74a806e9bc89a45e2b4ab59080b72c23854fa595ccd68409165a, cannot remove - tried to kill container, but did not receive an exit event\n"}
2022-11-07T16:52:00.398-0500 V4 Task finished {"task_name": "delete-kind-cluster", "duration": "25.31542549s"}
2022-11-07T16:52:00.398-0500 V4 ----------------------------------
2022-11-07T16:52:00.398-0500 V4 Saving checkpoint {"file": "baseline-test-checkpoint.yaml"}
2022-11-07T16:52:00.483-0500 V4 Tasks completed {"duration": "2h9m8.135509472s"}
2022-11-07T16:52:00.487-0500 V3 Cleaning up long running container {"name": "eksa_1667850170484856000"}
2022-11-07T16:52:00.487-0500 V6 Executing command {"cmd": "/usr/local/bin/docker rm -f -v eksa_1667850170484856000"}
Error: failed to upgrade cluster: executing delete cluster: Deleting cluster "baseline-test-eks-a-cluster" ...
ERROR: failed to delete cluster "baseline-test-eks-a-cluster": failed to delete nodes: command "docker rm -f -v baseline-test-eks-a-cluster-control-plane" failed with error: exit status 1
Command Output: Error response from daemon: Could not kill running container 43583218989e74a806e9bc89a45e2b4ab59080b72c23854fa595ccd68409165a, cannot remove - tried to kill container, but did not receive an exit event
```
I then manually retried deleting the kind cluster and it succeeded
**What you expected to happen**:
I would expect this kind cluster deletion to succeed on the first try, or else retry several times
**How to reproduce it (as minimally and precisely as possible)**:
Upgrade a management cluster managing many EKS-A/CAPI resources
**Anything else we need to know?**:
**Environment**:
- EKS Anywhere Release: 0.12.0 to 0.12.1 with CloudStack provider
- EKS Distro Release:
Contributor guide
Research direction
Start at the upgrade workflow's `delete-kind-cluster` task and the `kind delete cluster` command shown in the logs. Reproduce an upgrade of a management cluster with many EKS-A/CAPI resources, then trace the deletion failure and retry behavior. Done means the kind cluster deletion succeeds on the first try or is retried until it succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, kubernetes
- Domain
- cli, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100