aws / aws/eks-anywhere

eksctl anywhere upgrade leaves drivers/controllers in broken state

Open
#896 7 comments 0 reactions 0 assignees View on GitHub
external
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
After running a cluster upgrade command (where in my case the worker nodes were all replaced), the vsphere-csi-controller has started throwing errors:
```
Error processing "csi-7f6a00400ffb52421987f76ade13928a8e3f5582144chd03a921e4b5b6b2bb30": failed to detach: rpc error: code = Internal desc = failed to find VirtualMachine for node:"dev-md-0-6f5f5c955-rghzs". Error: node wasn't fo
```
However, I can confirm that the PersistentVolumes successfully re-attached to the new worker nodes. My persistent workloads have no data loss post-upgrade.

Same with capi-controller-manager:
```
E1230 16:46:14.917830 1 machine_controller.go:685] controllers/Machine "msg"="Unable to retrieve machine from node" "error"="no matching Machine" "node"="dev-md-0-5b6bd949cd-qzpxs"
E1230 16:46:14.917864 1 machine_controller.go:685] controllers/Machine "msg"="Unable to retrieve machine from node" "error"="no matching Machine" "node"="dev-md-0-5b6bd949cd-qzpxs"
E1230 16:46:55.067653 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="could not find infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereMachine \"dev-worker-node-template-1640730669596-qsnqj\" for Machine \"dev-md-0-5b6bd949cd-xwnbv\" in namespace \"eksa-system\", requeuing: requeue in 30s" "controller"="machine" "name"="dev-md-0-5b6bd949cd-xwnbv" "namespace"="eksa-system"
E1230 16:47:10.156694 1 leaderelection.go:331] error retrieving resource lock capi-system/controller-leader-election-capi: etcdserver: leader changed
E1230 16:47:10.345013 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="could not find infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereMachine \"dev-worker-node-template-1640730669596-5rzbc\" for Machine \"dev-md-0-5b6bd949cd-bcwxw\" in namespace \"eksa-system\", requeuing: requeue in 30s" "controller"="machine" "name"="dev-md-0-5b6bd949cd-bcwxw" "namespace"="eksa-system"
I1230 16:47:11.602700
```

This seems to result in a number of services experiencing errors like:
```
Error: error running manager: leader election lost
```
Seeing this with:
- kube-vip-cloud-provider
- etcadm-bootstrap-provider-controller
- capi-kubeadm-control-plane-controller
- eks-controller-manager
- etc (looks like anything that does leader election)

It seems that the old worker nodes are still being referenced.

**What you expected to happen**:
References to old worker nodes to be cleaned up - including VolumeAttachments

**How to reproduce it (as minimally and precisely as possible)**:
1. Create a cluster
2. Assign a PersistentVolume to a workload
3. Update the EKS Anywhere cluster spec to a new template (or something that would trigger a node replacement)
4. Post upgrade - observe the errors in vsphere-csi-driver

**Anything else we need to know?**:
This seems to be similar to https://github.com/kubernetes-csi/external-attacher/issues/215

**Environment**:
- EKS Anywhere Release:
- EKS Distro Release: v1.21.2-eks-1-21-5

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the cluster upgrade and node-replacement steps, then inspect the vsphere-csi-driver, capi-controller-manager, and other listed controller logs. Trace references to old worker nodes and VolumeAttachments, including the reported leader-election errors. Done means old-node references are cleaned up after upgrade and the affected controllers operate without repeated reconciliation or leader-election failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.