EKS Anywhere 0.25.3 upgrade stalls because kubeadm omits kubeadm:apiserver-kubelet-client ClusterRoleBinding on upgraded workload clusters
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
When upgrading a workload cluster with stacked etcd from EKS Anywhere `v0.24.4` to `v0.25.3` (Kubernetes `v1.34` → `v1.35`), the upgrade process gets stuck when rolling out new control plane nodes.
All control plane `Machine`s report `Unknown` status
```
$ kubectl get ma -A -l cluster.x-k8s.io/cluster-name=eksa-w01
NAMESPACE NAME CLUSTER NODE NAME READY AVAILABLE UP-TO-DATE PHASE AGE VERSION
eksa-system eksa-w01-cppng eksa-w01 eksa-w01-cppng Unknown False False Running 69m v1.34.3-eks-1-34-13
eksa-system eksa-w01-md-0-8gpmr-crglq eksa-w01 eksa-w01-md-0-8gpmr-crglq True True True Running 69m v1.34.3-eks-1-34-13
eksa-system eksa-w01-md-0-8gpmr-shfx7 eksa-w01 eksa-w01-md-0-8gpmr-shfx7 True True True Running 69m v1.34.3-eks-1-34-13
eksa-system eksa-w01-tn5dz eksa-w01 eksa-w01-tn5dz Unknown False False Running 67m v1.34.3-eks-1-34-13
eksa-system eksa-w01-zzgfb eksa-w01 eksa-w01-zzgfb Unknown False True Running 2m48s v1.35.6-eks-1-35-11
```
with condition:
```yaml
- lastTransitionTime: "2026-08-17T15:28:28Z"
message: '* EtcdMemberHealthy: Failed to connect to etcd: failed to get etcd status:
context deadline exceeded'
observedGeneration: 3
reason: ReadyUnknown
status: Unknown
type: Ready
```
Simultaneously, I can see in the `capi-kubeadm-control-plane-controller-manager` logs
```json
{
"level": "warn",
"ts": "2026-08-17T15:35:35.672791Z",
"caller": "v3@v3.6.6/retry_interceptor.go:65",
"msg": "retrying of unary invoker failed",
"target": "etcd-endpoints://0xc00141c5a0/etcd-eksa-w01-cppng",
"method": "/etcdserverpb.Maintenance/Status",
"attempt": 0,
"error": "rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: error upgrading connection: unable to upgrade connection: Forbidden (user=kube-apiserver-kubelet-client, verb=create, resource=nodes, subresource(s)=[proxy])\""
}
```
with the authorization issue clearly visible.
**Root Cause Analysis**
During upgrade to Kubernetes 1.35, `apiserver-kubelet-client.crt` no longer contains group `kubeadm:cluster-admins`:
```
root@eksa-w01-zzgfb:~# openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -subject
subject=CN = kube-apiserver-kubelet-client
```
(previously it was `subject=O = kubeadm:cluster-admins, CN = kube-apiserver-kubelet-client`).
Existing upgraded clusters therefore lose authorization through the old `kubeadm:cluster-admins binding` (note: the Ubuntu image built for Kubernetes v1.35.2 and EKS Anywhere v0.25.2 is not afected by this). As a result, the `kube-apiserver-kubelet-client` user used by the KubeadmControlPlane controller health checks loses the ability to tunnel the calls through API server proxy:
```
$ kubectl auth can-i create nodes --subresource proxy --as=kube-apiserver-kubelet-client --all-namespaces
no
```
Fresh 1.35 clusters receive `kubeadm:apiserver-kubelet-client` `ClusterRoleBinding`, which directly binds the user to `system:kubelet-api-admin`, so the above command returns `yes`.
This might be related to a Cluster API issue: https://github.com/kubernetes-sigs/cluster-api/issues/13770
**What you expected to happen**:
Upgrading an existing workload cluster must create or reconcile `kubeadm:apiserver-kubelet-client` before control-plane health checks run.
**How to reproduce it (as minimally and precisely as possible)**:
1. Deploy an EKS Anywhere management cluster on **v0.24.4**.
2. Create a workload cluster with **stacked etcd** (3 control plane nodes).
3. Upgrade the management cluster to **v0.25.3**.
4. Upgrade the workload cluster spec to **v0.25.3** (Kubernetes **v1.35**).
5. Observe the upgrade stalling while rolling out new control plane nodes.
**Anything else we need to know?**:
Applying the missing `ClusterRoleBinding` directly on the **workload cluster** immediately unblocks CAPI reconciliation and allows the upgrade to finish:
```bash
$ kubectl --kubeconfig apply -f - <
Contributor guide
Research direction
Start by tracing the workload-cluster upgrade and kubeadm control-plane reconciliation paths that create or reconcile RBAC objects; the issue provides no file or test names. Reproduce the v0.24.4-to-v0.25.3 stacked-etcd upgrade, verify the binding exists before health checks, and confirm reconciliation completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100