Workload cluster upgrade fails when adding new control plane nodes with stacked etcd during GitOps upgrades (v0.23.3→v0.24.6 and v0.24.6→v0.25.2)

## What happened

We are running EKS Anywhere on vSphere with a dedicated management cluster and multiple workload clusters managed through GitOps.

Our upgrade path was:

- EKS Anywhere v0.23.3 + Kubernetes v1.33 → EKS Anywhere v0.24.6 + Kubernetes v1.34
- EKS Anywhere v0.24.6 + Kubernetes v1.34 → EKS Anywhere v0.25.2 + Kubernetes v1.35

We followed the upgrade strategy recommended in the documentation:

1. Upgrade one EKS Anywhere minor version at a time.
2. Upgrade one Kubernetes minor version at a time.
3. Upgrade the management cluster first.
4. Upgrade workload clusters afterwards.

The management cluster was upgraded using the CLI tool:

```bash
eksctl anywhere upgrade cluster
```

Workload clusters were upgraded through GitOps by updating the cluster specifications.

During both upgrade cycles, some workload clusters failed while replacing control plane nodes.

The failure occurs after the new VM is provisioned and after the node has already joined the cluster, but before it successfully completes the control plane join workflow.

The error observed on the new control plane node is:

```text
Error running bootstrapper cmd: error joining controlplane:
Error running command:
/opt/bin/kubeadm -v9 join --skip-phases preflight,control-plane-prepare,kubelet-start,control-plane-join/etcd,kubelet-wait-bootstrap --config /tmp/kubeadm-join-config.yaml:
exit status 1
```

The upgrade then becomes blocked waiting for the new control plane node to become ready.

---

## What you expected to happen

The newly provisioned control plane node should successfully complete the `kubeadm join` control plane workflow and become a healthy control plane member automatically, allowing the rolling upgrade to continue without manual intervention.

---

## How to reproduce it (as minimally and precisely as possible)

Environment:

- Provider: vSphere
- Management cluster upgraded via CLI
- Workload clusters upgraded through GitOps
- Bottlerocket nodes

Observed behavior across clusters:

| Cluster | Topology | Upgrade Method | Result |
|----------|-----------|----------------|---------|
| Management Cluster | 3 Control Planes + 3 External ETCD + 3 Workers | CLI | Success |
| Workload w-01 | 3 Control Planes + 3 External ETCD + 3 Workers | GitOps | Success |
| Workload w-02 | 3 Control Planes (stacked/integrated etcd) + 3 Workers | GitOps | Failed |
| Workload w-03 | 1 Control Plane (stacked/integrated etcd) + 3 Workers | GitOps | Failed |

The issue appears to affect only clusters using **stacked/integrated etcd**.

We have not observed this problem on clusters using **external ETCD**.

The same behavior was reproduced during both upgrade paths:

- v0.23.3 / Kubernetes 1.33 → v0.24.6 / Kubernetes 1.34
- v0.24.6 / Kubernetes 1.34 → v0.25.2 / Kubernetes 1.35

---

## Anything else we need to know?

We were able to recover and complete the upgrade manually.

The newly created node had already joined the cluster as a Kubernetes node but never completed the control plane bootstrap process.

The following procedure consistently allowed us to recover the failed control plane node and complete the upgrade.

### 1. SSH into the new control plane node

```bash
ssh -i ~/.ssh/eksa-ed25519 ec2-user@172.16.119.116

sudo sheltie
```

### 2. Wait until the kubeadm bootstrap container fails

```bash
systemctl is-failed host-containers@kubeadm-bootstrap.service
```

Wait until the service reports:

```text
failed
```

### 3. Retrieve the kubeadm-bootstrap container image

```bash
IMAGE_ID=$(apiclient get | apiclient exec admin jq -r '.settings["host-containers"]["kubeadm-bootstrap"].source')

ctr image pull ${IMAGE_ID}
```

### 4. Generate a new bootstrap token

The following command prints a valid join command that will be useful later to obtain:

- API Server endpoint
- Bootstrap token
- CA certificate hash

```bash
ctr run \
--net-host \
--mount type=bind,src=/var/lib/kubeadm,dst=/var/lib/kubeadm,options=rbind:rw \
--mount type=bind,src=/etc/kubernetes,dst=/etc/kubernetes,options=rbind:rw \
--mount type=bind,src=/tmp,dst=/tmp,options=rbind:rw \
--mount type=bind,src=/proc,dst=/proc,options=rbind:ro \
--rm ${IMAGE_ID} tmp-kubeadm-token-create \
/opt/bin/kubeadm token create \
--ttl 2h \
--print-join-command \
--kubeconfig /var/lib/kubeadm/admin.conf
```

Example output:

```bash
kubeadm join 172.16.119.16:6443 \
--token 1435b7.2iexxxxxxxxxxxxxx \
--discovery-token-ca-cert-hash sha256:3c90fxxxxxxxxxxxxxxxxxxxxxxxx
```

### 5. Obtain node-specific values

```bash
apiclient get settings.network.hostname

apiclient get settings.kubernetes.node-ip
```

### 6. Create a JoinConfiguration

Using the values collected previously, create:

```bash
cat </tmp/kubeadm-join-config.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: JoinConfiguration
discovery:
bootstrapToken:
apiServerEndpoint: 172.16.119.16:6443
caCertHashes:
- sha256:3c90f...
token: 1435b7.2ie...
nodeRegistration:
criSocket: unix:///var/run/containerd/containerd.sock
imagePullPolicy: IfNotPresent
kubeletExtraArgs:
- name: anonymous-auth
value: "false"
- name: cloud-provider
value: external
- name: read-only-port
value: "0"
- name: tls-cipher-suites
value: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
name: w01-kdl2q
taints: null
controlPlane:
localAPIEndpoint:
advertiseAddress: 172.16.119.116
bindPort: 6443
EOF
```

### 7. Create the admin.conf symlink

```bash
ln -sf /var/lib/kubeadm/admin.conf /etc/kubernetes/admin.conf
```

### 8. Execute the control plane join manually

```bash
ctr run \
--net-host \
--mount type=bind,src=/var/lib/kubeadm,dst=/var/lib/kubeadm,options=rbind:rw \
--mount type=bind,src=/var/lib/etcd,dst=/var/lib/etcd,options=rbind:rw \
--mount type=bind,src=/etc/kubernetes,dst=/etc/kubernetes,options=rbind:rw \
--mount type=bind,src=/etc/kubernetes/manifests,dst=/etc/kubernetes/manifests,options=rbind:rw \
--mount type=bind,src=/tmp,dst=/tmp,options=rbind:rw \
--mount type=bind,src=/proc,dst=/proc,options=rbind:ro \
--rm ${IMAGE_ID} tmp-cert-renew \
/opt/bin/kubeadm -v9 join \
--skip-phases preflight,check-etcd,control-plane-prepare,kubelet-start,etcd-join,wait-control-plane,kubelet-wait-bootstrap \
--config /tmp/kubeadm-join-config.yaml
```

After executing the command above, the node successfully joined as a control plane member and the upgrade process completed normally.

### Additional notes

I am not a Cluster API expert, so I cannot guarantee that the selected `--skip-phases` list is the most appropriate one.

These are simply the phases that consistently worked for us across multiple clusters and during two separate upgrade cycles.

It is possible that there is a cleaner or more appropriate recovery procedure.

However, the fact that the issue only appears on clusters using stacked/integrated etcd and can be consistently bypassed through a manual `kubeadm join` makes me suspect there may be a problem in the automated control plane bootstrap/join workflow during workload cluster upgrades.

If someone encounters the same issue, I would recommend validating the proposed workaround in a non-production environment first.

---

## Environment

- Infrastructure Provider: vSphere
- OS: Bottlerocket

### Upgrade Path #1

- EKS Anywhere Release:
- Source: v0.23.3
- Target: v0.24.6

- EKS Distro Release:
- Kubernetes v1.33
- Kubernetes v1.34

### Upgrade Path #2

- EKS Anywhere Release:
- Source: v0.24.6
- Target: v0.25.2

- EKS Distro Release:
- Kubernetes v1.34
- Kubernetes v1.35

Start by tracing the kubeadm-bootstrap.service and automated control plane join workflow used during GitOps workload-cluster upgrades, comparing stacked/integrated etcd with external etcd. Reproduce the failure on a non-production stacked-etcd cluster and inspect the failed kubeadm join output. Done means new control plane nodes complete the join automatically and the rolling upgrade proceeds without manual recovery.

Written by the indexing model from the issue text.