aws / aws/eks-anywhere

EKS Anywhere 0.25.3 upgrade stalls because kubeadm omits kubeadm:apiserver-kubelet-client ClusterRoleBinding on upgraded workload clusters

Open
#10,941 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:

When upgrading a workload cluster with stacked etcd from EKS Anywhere `v0.24.4` to `v0.25.3` (Kubernetes `v1.34` → `v1.35`), the upgrade process gets stuck when rolling out new control plane nodes.

All control plane `Machine`s report `Unknown` status

```
$ kubectl get ma -A -l cluster.x-k8s.io/cluster-name=eksa-w01
NAMESPACE NAME CLUSTER NODE NAME READY AVAILABLE UP-TO-DATE PHASE AGE VERSION
eksa-system eksa-w01-cppng eksa-w01 eksa-w01-cppng Unknown False False Running 69m v1.34.3-eks-1-34-13
eksa-system eksa-w01-md-0-8gpmr-crglq eksa-w01 eksa-w01-md-0-8gpmr-crglq True True True Running 69m v1.34.3-eks-1-34-13
eksa-system eksa-w01-md-0-8gpmr-shfx7 eksa-w01 eksa-w01-md-0-8gpmr-shfx7 True True True Running 69m v1.34.3-eks-1-34-13
eksa-system eksa-w01-tn5dz eksa-w01 eksa-w01-tn5dz Unknown False False Running 67m v1.34.3-eks-1-34-13
eksa-system eksa-w01-zzgfb eksa-w01 eksa-w01-zzgfb Unknown False True Running 2m48s v1.35.6-eks-1-35-11
```

with condition:

```yaml
- lastTransitionTime: "2026-08-17T15:28:28Z"
message: '* EtcdMemberHealthy: Failed to connect to etcd: failed to get etcd status:
context deadline exceeded'
observedGeneration: 3
reason: ReadyUnknown
status: Unknown
type: Ready
```

Simultaneously, I can see in the `capi-kubeadm-control-plane-controller-manager` logs
```json
{
"level": "warn",
"ts": "2026-08-17T15:35:35.672791Z",
"caller": "v3@v3.6.6/retry_interceptor.go:65",
"msg": "retrying of unary invoker failed",
"target": "etcd-endpoints://0xc00141c5a0/etcd-eksa-w01-cppng",
"method": "/etcdserverpb.Maintenance/Status",
"attempt": 0,
"error": "rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: error upgrading connection: unable to upgrade connection: Forbidden (user=kube-apiserver-kubelet-client, verb=create, resource=nodes, subresource(s)=[proxy])\""
}
```
with the authorization issue clearly visible.

**Root Cause Analysis**

During upgrade to Kubernetes 1.35, `apiserver-kubelet-client.crt` no longer contains group `kubeadm:cluster-admins`:

```
root@eksa-w01-zzgfb:~# openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -subject
subject=CN = kube-apiserver-kubelet-client
```

(previously it was `subject=O = kubeadm:cluster-admins, CN = kube-apiserver-kubelet-client`).

Existing upgraded clusters therefore lose authorization through the old `kubeadm:cluster-admins binding` (note: the Ubuntu image built for Kubernetes v1.35.2 and EKS Anywhere v0.25.2 is not afected by this). As a result, the `kube-apiserver-kubelet-client` user used by the KubeadmControlPlane controller health checks loses the ability to tunnel the calls through API server proxy:

```
$ kubectl auth can-i create nodes --subresource proxy --as=kube-apiserver-kubelet-client --all-namespaces
no
```

Fresh 1.35 clusters receive `kubeadm:apiserver-kubelet-client` `ClusterRoleBinding`, which directly binds the user to `system:kubelet-api-admin`, so the above command returns `yes`.

This might be related to a Cluster API issue: https://github.com/kubernetes-sigs/cluster-api/issues/13770

**What you expected to happen**:

Upgrading an existing workload cluster must create or reconcile `kubeadm:apiserver-kubelet-client` before control-plane health checks run.

**How to reproduce it (as minimally and precisely as possible)**:

1. Deploy an EKS Anywhere management cluster on **v0.24.4**.
2. Create a workload cluster with **stacked etcd** (3 control plane nodes).
3. Upgrade the management cluster to **v0.25.3**.
4. Upgrade the workload cluster spec to **v0.25.3** (Kubernetes **v1.35**).
5. Observe the upgrade stalling while rolling out new control plane nodes.

**Anything else we need to know?**:

Applying the missing `ClusterRoleBinding` directly on the **workload cluster** immediately unblocks CAPI reconciliation and allows the upgrade to finish:

```bash
$ kubectl --kubeconfig apply -f - <

Contributor guide

Open the contributing guide

Research direction

Start by tracing the workload-cluster upgrade and kubeadm control-plane reconciliation paths that create or reconcile RBAC objects; the issue provides no file or test names. Reproduce the v0.24.4-to-v0.25.3 stacked-etcd upgrade, verify the binding exists before health checks, and confirm reconciliation completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.