eksctl anywhere upgrade leaves drivers/controllers in broken state
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
After running a cluster upgrade command (where in my case the worker nodes were all replaced), the vsphere-csi-controller has started throwing errors:
```
Error processing "csi-7f6a00400ffb52421987f76ade13928a8e3f5582144chd03a921e4b5b6b2bb30": failed to detach: rpc error: code = Internal desc = failed to find VirtualMachine for node:"dev-md-0-6f5f5c955-rghzs". Error: node wasn't fo
```
However, I can confirm that the PersistentVolumes successfully re-attached to the new worker nodes. My persistent workloads have no data loss post-upgrade.
Same with capi-controller-manager:
```
E1230 16:46:14.917830 1 machine_controller.go:685] controllers/Machine "msg"="Unable to retrieve machine from node" "error"="no matching Machine" "node"="dev-md-0-5b6bd949cd-qzpxs"
E1230 16:46:14.917864 1 machine_controller.go:685] controllers/Machine "msg"="Unable to retrieve machine from node" "error"="no matching Machine" "node"="dev-md-0-5b6bd949cd-qzpxs"
E1230 16:46:55.067653 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="could not find infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereMachine \"dev-worker-node-template-1640730669596-qsnqj\" for Machine \"dev-md-0-5b6bd949cd-xwnbv\" in namespace \"eksa-system\", requeuing: requeue in 30s" "controller"="machine" "name"="dev-md-0-5b6bd949cd-xwnbv" "namespace"="eksa-system"
E1230 16:47:10.156694 1 leaderelection.go:331] error retrieving resource lock capi-system/controller-leader-election-capi: etcdserver: leader changed
E1230 16:47:10.345013 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="could not find infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereMachine \"dev-worker-node-template-1640730669596-5rzbc\" for Machine \"dev-md-0-5b6bd949cd-bcwxw\" in namespace \"eksa-system\", requeuing: requeue in 30s" "controller"="machine" "name"="dev-md-0-5b6bd949cd-bcwxw" "namespace"="eksa-system"
I1230 16:47:11.602700
```
This seems to result in a number of services experiencing errors like:
```
Error: error running manager: leader election lost
```
Seeing this with:
- kube-vip-cloud-provider
- etcadm-bootstrap-provider-controller
- capi-kubeadm-control-plane-controller
- eks-controller-manager
- etc (looks like anything that does leader election)
It seems that the old worker nodes are still being referenced.
**What you expected to happen**:
References to old worker nodes to be cleaned up - including VolumeAttachments
**How to reproduce it (as minimally and precisely as possible)**:
1. Create a cluster
2. Assign a PersistentVolume to a workload
3. Update the EKS Anywhere cluster spec to a new template (or something that would trigger a node replacement)
4. Post upgrade - observe the errors in vsphere-csi-driver
**Anything else we need to know?**:
This seems to be similar to https://github.com/kubernetes-csi/external-attacher/issues/215
**Environment**:
- EKS Anywhere Release:
- EKS Distro Release: v1.21.2-eks-1-21-5
Contributor guide
Research direction
Start by reproducing the cluster upgrade and node-replacement steps, then inspect the vsphere-csi-driver, capi-controller-manager, and other listed controller logs. Trace references to old worker nodes and VolumeAttachments, including the reported leader-election errors. Done means old-node references are cleaned up after upgrade and the affected controllers operate without repeated reconciliation or leader-election failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100