Extra CP node created when provisioning a Snow BR unstacked etcd cluster

When performing a version upgrade on a management cluster through the cli using Bottlerocket, an extra cp node gets provisioned and joins the cluster before scaling back down to the set number of nodes in the spec. When investigating why this is the case, it seems like the very first cp node that gets provisioned has a failure in the kubelet logs, cause that machine to be replaced by the newer one after the other cp nodes also rollout. We want to pinpoint how we can have the first node work without issue as it is not an issue with the other providers.

One behavior that is different from the other providers is that a cp node gets provisioned and goes to running state before the etcd machines finishes upgrading, which in something like vsphere, etcd machines fully roll out first before a cp node gets provisioned and goes into the running state.

Kubelet logs (saw variations of network related things):
```
Jan 20 22:36:21 koppv-control-plane-1-gpn2w kubelet[2090]: E0120 22:36:21.111647 2090 kubelet.go:2391] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Jan 20 22:36:26 koppv-control-plane-1-gpn2w kubelet[2090]: E0120 22:36:26.113901 2090 kubelet.go:2391] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Jan 20 22:36:31 koppv-control-plane-1-gpn2w kubelet[2090]: E0120 22:36:31.115536 2090 kubelet.go:2391] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Jan 20 22:39:39 koppv-control-plane-1-gpn2w kubelet[2090]: E0120 22:39:39.350626 2090 controller.go:187] failed to update lease, error: Put "https://192.168.1.48:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/koppv-control-plane-1-gpn2w?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Jan 20 22:39:45 koppv-control-plane-1-gpn2w kubelet[2090]: E0120 22:39:45.975464 2090 kubelet_node_status.go:460] "Error updating node status, will retry" err="error getting node \"koppv-control-plane-1-gpn2w\": Get \"https://192.168.1.48:6443/api/v1/nodes/koppv-control-plane-1-gpn2w?resourceVersion=0&timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Jan 20 22:39:49 koppv-control-plane-1-gpn2w kubelet[2090]: E0120 22:39:49.351716 2090 controller.go:187] failed to update lease, error: Put "https://192.168.1.48:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/koppv-control-plane-1-gpn2w?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Jan 20 22:39:55 koppv-control-plane-1-gpn2w kubelet[2090]: E0120 22:39:55.976417 2090 kubelet_node_status.go:460] "Error updating node status, will retry" err="error getting node \"koppv-control-plane-1-gpn2w\": Get \"https://192.168.1.48:6443/api/v1/nodes/koppv-control-plane-1-gpn2w?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Jan 20 22:39:57 koppv-control-plane-1-gpn2w kubelet[2090]: E0120 22:39:57.652340 2090 controller.go:187] failed to update lease, error: Put "https://192.168.1.48:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/koppv-control-plane-1-gpn2w?timeout=10s": read tcp 192.168.1.38:39032->192.168.1.48:6443: read: connection reset by peer
Jan 20 22:43:21 koppv-control-plane-1-gpn2w kubelet[2090]: E0120 22:43:21.963732 2090 controller.go:187] failed to update lease, error: Put "https://192.168.1.48:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/koppv-control-plane-1-gpn2w?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Jan 20 22:43:22 koppv-control-plane-1-gpn2w kubelet[2090]: E0120 22:43:22.651367 2090 kubelet_node_status.go:460] "Error updating node status, will retry" err="error getting node \"koppv-control-plane-1-gpn2w\": Get \"https://192.168.1.48:6443/api/v1/nodes/koppv-control-plane-1-gpn2w?resourceVersion=0&timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Jan 20 22:43:26 koppv-control-plane-1-gpn2w kubelet[2090]: E0120 22:43:26.032990 2090 controller.go:187] failed to update lease, error: Put "https://192.168.1.48:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/koppv-control-plane-1-gpn2w?timeout=10s": read tcp 192.168.1.38:35518->192.168.1.48:6443: read: connection reset by peer
```
```
Jan 25 00:40:31 koppv-control-plane-1-xxzjt kubelet[1614]: E0125 00:40:31.591074 1614 kubelet.go:2349] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Jan 25 00:46:45 koppv-control-plane-1-xxzjt kubelet[1614]: E0125 00:46:45.037139 1614 kubelet_node_status.go:460] "Error updating node status, will retry" err="error getting node \"koppv-control-plane-1-xxzjt\": Get \"https://192.168.1.48:6443/api/v1/nodes/koppv-control-plane-1-xxzjt?resourceVersion=0&timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Jan 25 00:46:45 koppv-control-plane-1-xxzjt kubelet[1614]: E0125 00:46:45.692271 1614 controller.go:187] failed to update lease, error: Put "https://192.168.1.48:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/koppv-control-plane-1-xxzjt?timeout=10s": context deadline exceeded
Jan 25 00:46:48 koppv-control-plane-1-xxzjt kubelet[1614]: E0125 00:46:48.980858 1614 controller.go:187] failed to update lease, error: Put "https://192.168.1.48:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/koppv-control-plane-1-xxzjt?timeout=10s": read tcp 192.168.1.25:46132->192.168.1.48:6443: read: connection reset by peer
```

Some additional logs from controllers during upgrade that might be worth checking, but nothing stands out here:
```
I0120 08:50:29.341643 1 scale.go:212] "msg"="Waiting for control plane to pass preflight checks" "cluster-name"="koppv" "name"="koppv" "namespace"="eksa-system" "failures"="machine koppv-gxks2 reports APIServerPodHealthy condition is false (Info, Waiting for startup or readiness probes)"
I0120 08:50:29.377881 1 controller.go:299] "Reconcile KubeadmControlPlane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=6131eae0-5e45-43ac-a991-70bd71d38171 cluster="koppv"
I0120 08:50:31.676399 1 controller.go:392] "Rolling out Control Plane machines" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=6131eae0-5e45-43ac-a991-70bd71d38171 cluster="koppv" needRollout=[koppv-2k8dh]
I0120 08:50:31.819125 1 controller.go:299] "Reconcile KubeadmControlPlane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=99db5510-3562-4d75-b8a9-7be8d2e468b9 cluster="koppv"
I0120 08:50:33.478877 1 controller.go:410] "Adding upgrade complete annotation on etcdadmCluster" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=99db5510-3562-4d75-b8a9-7be8d2e468b9 cluster="koppv"
I0120 08:50:33.520772 1 controller.go:441] "Scaling down control plane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=99db5510-3562-4d75-b8a9-7be8d2e468b9 cluster="koppv" Desired=3 Existing=4
I0120 08:50:33.520847 1 scale.go:174] "msg"="Waiting for machines to be deleted" "cluster-name"="koppv" "name"="koppv" "namespace"="eksa-system" "Machines"="koppv-2k8dh"
I0120 08:50:33.593418 1 controller.go:299] "Reconcile KubeadmControlPlane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=e45cbcaa-6999-45a4-8c1a-35ef83b932e5 cluster="koppv"
I0120 08:50:35.265012 1 controller.go:441] "Scaling down control plane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=e45cbcaa-6999-45a4-8c1a-35ef83b932e5 cluster="koppv" Desired=3 Existing=4
I0120 08:50:35.265064 1 scale.go:174] "msg"="Waiting for machines to be deleted" "cluster-name"="koppv" "name"="koppv" "namespace"="eksa-system" "Machines"="koppv-2k8dh"
I0120 08:50:35.346683 1 controller.go:299] "Reconcile KubeadmControlPlane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=06c6b87a-7c6c-4a01-b207-65694b447e08 cluster="koppv"
I0120 08:50:37.037747 1 controller.go:441] "Scaling down control plane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=06c6b87a-7c6c-4a01-b207-65694b447e08 cluster="koppv" Desired=3 Existing=4
I0120 08:50:37.037813 1 scale.go:174] "msg"="Waiting for machines to be deleted" "cluster-name"="koppv" "name"="koppv" "namespace"="eksa-system" "Machines"="koppv-2k8dh"
I0120 08:51:03.593745 1 controller.go:299] "Reconcile KubeadmControlPlane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=1d31c98a-e404-4a75-848c-18d7f375d395 cluster="koppv"
I0120 08:51:03.717627 1 controller.go:441] "Scaling down control plane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=1d31c98a-e404-4a75-848c-18d7f375d395 cluster="koppv" Desired=3 Existing=4
I0120 08:51:03.717679 1 scale.go:174] "msg"="Waiting for machines to be deleted" "cluster-name"="koppv" "name"="koppv" "namespace"="eksa-system" "Machines"="koppv-2k8dh"
I0120 08:51:07.131727 1 controller.go:299] "Reconcile KubeadmControlPlane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=9f81d65c-1de8-420f-9976-ec1bbdeac3e2 cluster="koppv"
I0120 08:51:07.248253 1 controller.go:441] "Scaling down control plane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=9f81d65c-1de8-420f-9976-ec1bbdeac3e2 cluster="koppv" Desired=3 Existing=4
I0120 08:51:07.248308 1 scale.go:174] "msg"="Waiting for machines to be deleted" "cluster-name"="koppv" "name"="koppv" "namespace"="eksa-system" "Machines"="koppv-2k8dh"
I0120 08:51:07.320234 1 controller.go:299] "Reconcile KubeadmControlPlane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=c2c0bb99-68c1-40b4-97c8-0c6a77749121 cluster="koppv"
E0120 08:51:08.870661 1 controller.go:326] "Reconciler error" err="failed to patch machine koppv-2k8dh: [Machine.cluster.x-k8s.io \"koppv-2k8dh\" not found, machines.cluster.x-k8s.io \"koppv-2k8dh\" not found]" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=c2c0bb99-68c1-40b4-97c8-0c6a77749121
```

```
I0120 08:50:32.335296 1 awssnowmachine_controller.go:476] "msg"="static ip configured" "aWSSnowMachine"={"name":"koppv-control-plane-1-v796q","namespace":"eksa-system"}
"cluster"="koppv" "controller"="awssnowmachine" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="AWSSnowMachine" "machine"="koppv-gxks2" "name"="koppv-cont
rol-plane-1-v796q" "namespace"="eksa-system" "reconcileID"="5797ce8f-3c2f-47b5-b719-c887f6aaf66d" "192.168.1.12"="InternalIP"
I0120 08:50:32.336533 1 cluster.go:153] "msg"="Machine koppv-control-plane-1-4dljj already assigned to device with IP: 192.168.1.76" "aWSSnowMachine"={"name":"koppv-cont
rol-plane-1-4dljj","namespace":"eksa-system"} "cluster"="koppv" "controller"="awssnowmachine" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="AWSSnowMachi
ne" "machine"="koppv-2k8dh" "name"="koppv-control-plane-1-4dljj" "namespace"="eksa-system" "reconcileID"="5641fb1a-92a0-4d24-ac77-456028b84def"
I0120 08:50:32.337440 1 awssnowmachine_controller.go:497] "msg"="Handling deleted AWSSnowMachine" "aWSSnowMachine"={"name":"koppv-control-plane-1-4dljj","namespace":"eks
a-system"} "cluster"="koppv" "controller"="awssnowmachine" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="AWSSnowMachine" "machine"="koppv-2k8dh" "name"=
"koppv-control-plane-1-4dljj" "namespace"="eksa-system" "reconcileID"="5641fb1a-92a0-4d24-ac77-456028b84def"
I0120 08:50:32.360697 1 awssnowmachine_controller.go:544] "msg"="Terminating EC2 instance" "aWSSnowMachine"={"name":"koppv-control-plane-1-4dljj","namespace":"eksa-syste
m"} "cluster"="koppv" "controller"="awssnowmachine" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="AWSSnowMachine" "machine"="koppv-2k8dh" "name"="koppv-
control-plane-1-4dljj" "namespace"="eksa-system" "reconcileID"="5641fb1a-92a0-4d24-ac77-456028b84def" "instance-id"="s.i-846b5f206ccf70686"
I0120 08:50:32.360818 1 awssnowmachine_controller.go:553] "msg"="Deleting DNI of Instance ID:" "aWSSnowMachine"={"name":"koppv-control-plane-1-4dljj","namespace":"eksa-s
ystem"} "cluster"="koppv" "controller"="awssnowmachine" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="AWSSnowMachine" "machine"="koppv-2k8dh" "name"="ko
ppv-control-plane-1-4dljj" "namespace"="eksa-system" "reconcileID"="5641fb1a-92a0-4d24-ac77-456028b84def" "snowballDeviceSvc type:"={}
I0120 08:50:35.462312 1 awssnowmachine_controller.go:742] "msg"="vni has been deleted" "aWSSnowMachine"={"name":"koppv-control-plane-1-4dljj","namespace":"eksa-system"}
"cluster"="koppv" "controller"="awssnowmachine" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="AWSSnowMachine" "machine"="koppv-2k8dh" "name"="koppv-cont
rol-plane-1-4dljj" "namespace"="eksa-system" "reconcileID"="5641fb1a-92a0-4d24-ac77-456028b84def"
I0120 08:51:06.858760 1 awssnowmachine_controller.go:575] "msg"="EC2 instance successfully terminated" "aWSSnowMachine"={"name":"koppv-control-plane-1-4dljj","namespace"
:"eksa-system"} "cluster"="koppv" "controller"="awssnowmachine" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="AWSSnowMachine" "machine"="koppv-2k8dh" "n
ame"="koppv-control-plane-1-4dljj" "namespace"="eksa-system" "reconcileID"="5641fb1a-92a0-4d24-ac77-456028b84def" "instance-id"="s.i-846b5f206ccf70686"
```

```
I0120 08:50:31.766866 1 machine_controller.go:318] "Draining node" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-2k8dh" namespace="eksa-system" name="koppv-2k8dh" reconcileID=96619c33-3e04-4509-851a-4eb0f8e89a4d cluster="koppv" node="koppv-control-plane-1-4dljj"
I0120 08:50:31.779214 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=97be710a-087a-4f81-bc02-0d8c05d0476f
I0120 08:50:31.796224 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=5e25f518-a948-46ba-8fd4-6b4089ac37b5
I0120 08:50:31.834851 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=fe3489e1-2f95-4c36-8eaa-7cec2314e628
I0120 08:50:31.872947 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=2b659db3-f5d5-47e8-a0be-131f0bb0632e
E0120 08:50:31.885086 1 machine_controller.go:551] "WARNING: ignoring DaemonSet-managed Pods: kube-system/cilium-jxjqs, kube-system/kube-proxy-pswdm\n" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-2k8dh" namespace="eksa-system" name="koppv-2k8dh" reconcileID=96619c33-3e04-4509-851a-4eb0f8e89a4d cluster="koppv" node="koppv-control-plane-1-4dljj"
I0120 08:50:31.885149 1 machine_controller.go:572] "Drain successful" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-2k8dh" namespace="eksa-system" name="koppv-2k8dh" reconcileID=96619c33-3e04-4509-851a-4eb0f8e89a4d cluster="koppv" node="koppv-control-plane-1-4dljj"
I0120 08:50:31.903411 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=82fa8e50-2a94-43b0-8403-b161d26a6ec4
I0120 08:50:31.940599 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=ba655427-fab1-4ee0-ad3b-1f026bf19637
I0120 08:50:31.954482 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=9b4f7fa4-1d37-4cf2-8cb6-353ebc6672c8
I0120 08:50:31.959902 1 machine_controller.go:318] "Draining node" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-2k8dh" namespace="eksa-system" name="koppv-2k8dh" reconcileID=9ee81b54-2cfa-4f39-ba3a-dcb2820b7697 cluster="koppv" node="koppv-control-plane-1-4dljj"
E0120 08:50:31.998792 1 machine_controller.go:551] "WARNING: ignoring DaemonSet-managed Pods: kube-system/cilium-jxjqs, kube-system/kube-proxy-pswdm\n" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-2k8dh" namespace="eksa-system" name="koppv-2k8dh" reconcileID=9ee81b54-2cfa-4f39-ba3a-dcb2820b7697 cluster="koppv" node="koppv-control-plane-1-4dljj"
I0120 08:50:31.998860 1 machine_controller.go:572] "Drain successful" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-2k8dh" namespace="eksa-system" name="koppv-2k8dh" reconcileID=9ee81b54-2cfa-4f39-ba3a-dcb2820b7697 cluster="koppv" node="koppv-control-plane-1-4dljj"
I0120 08:50:33.450404 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=ab72fc2e-fe5c-496f-b118-099f4ea95949
I0120 08:50:33.459265 1 machine_controller.go:318] "Draining node" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-2k8dh" namespace="eksa-system" name="koppv-2k8dh" reconcileID=3df361e3-4563-4cb5-9e6a-efd564f38ffb cluster="koppv" node="koppv-control-plane-1-4dljj"
E0120 08:50:33.506365 1 machine_controller.go:551] "WARNING: ignoring DaemonSet-managed Pods: kube-system/cilium-jxjqs, kube-system/kube-proxy-pswdm\n" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-2k8dh" namespace="eksa-system" name="koppv-2k8dh" reconcileID=3df361e3-4563-4cb5-9e6a-efd564f38ffb cluster="koppv" node="koppv-control-plane-1-4dljj"
I0120 08:50:33.506405 1 machine_controller.go:572] "Drain successful" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-2k8dh" namespace="eksa-system" name="koppv-2k8dh" reconcileID=3df361e3-4563-4cb5-9e6a-efd564f38ffb cluster="koppv" node="koppv-control-plane-1-4dljj"
I0120 08:50:34.027207 1 machine_controller.go:296] "Deleting Kubernetes Node associated with Machine is not allowed" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-etcd-8f7cm" namespace="eksa-system" name="koppv-etcd-8f7cm" reconcileID=731158ae-70fe-4424-a405-9bc03eb76fac cluster="koppv" node="nil" cause="noderef is nil"
I0120 08:51:07.203543 1 machine_controller.go:390] "Deleting node" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-2k8dh" namespace="eksa-system" name="koppv-2k8dh" reconcileID=c5e971bb-679c-4ec3-bc33-7715f7e10e16 cluster="koppv" node="koppv-control-plane-1-4dljj"
I0120 08:51:07.216190 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=aa668a48-3ec2-41b4-8fc8-284afde8347d
I0120 08:51:07.217297 1 machinehealthcheck_controller.go:426] "Target has failed health check, marking for remediation" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=aa668a48-3ec2-41b4-8fc8-284afde8347d cluster="koppv" target="eksa-system/koppv-kcp-unhealthy/koppv-2k8dh/" reason="NodeNotFound" message=""
I0120 08:51:07.252124 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=adea9966-4911-4d8d-a56b-6300f97c0f14
I0120 08:51:07.253211 1 machinehealthcheck_controller.go:426] "Target has failed health check, marking for remediation" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=adea9966-4911-4d8d-a56b-6300f97c0f14 cluster="koppv" target="eksa-system/koppv-kcp-unhealthy/koppv-2k8dh/" reason="NodeNotFound" message=""
I0120 08:51:07.446149 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=18420589-de18-4680-b4b2-6c0c2001458d
I0120 08:51:07.447223 1 machinehealthcheck_controller.go:426] "Target has failed health check, marking for remediation" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=18420589-de18-4680-b4b2-6c0c2001458d cluster="koppv" target="eksa-system/koppv-kcp-unhealthy/koppv-2k8dh/" reason="NodeNotFound" message=""
I0120 08:51:07.472925 1 machinehealthcheck_controller.go:119] "Reconciling" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" machineHealthCheck="eksa-system/koppv-kcp-unhealthy" namespace="eksa-system" name="koppv-kcp-unhealthy" reconcileID=8a605d5e-0c70-48c3-855a-5eef2176b61f
E0120 08:51:07.477728 1 controller.go:326] "Reconciler error" err="machines.cluster.x-k8s.io \"koppv-2k8dh\" not found" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" machine="eksa-system/koppv-2k8dh" namespace="eksa-system" name="koppv-2k8dh" reconcileID=c5e971bb-679c-4ec3-bc33-7715f7e10e16
```
```
I0125 00:52:34.728998 1 controller.go:299] "Reconcile KubeadmControlPlane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmC
ontrolPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=c4634e9a-d396-4902-9af6-1e76b22fb6e9 cluster="koppv"
I0125 00:52:34.858414 1 controller.go:392] "Rolling out Control Plane machines" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="Kub
eadmControlPlane" kubeadmControlPlane="eksa-system/koppv" namespace="eksa-system" name="koppv" reconcileID=c4634e9a-d396-4902-9af6-1e76b22fb6e9 cluster="koppv" needRollout=[koppv-tndfp]
I0125 00:52:34.928205 1 scale.go:212] "msg"="Waiting for control plane to pass preflight checks" "cluster-name"="koppv" "name"="koppv" "namespace"="eksa-system" "failures"="[machi
ne koppv-964xd reports APIServerPodHealthy condition is false (Error, Missing node), machine koppv-964xd reports ControllerManagerPodHealthy condition is false (Error, Missing node), mac
hine koppv-964xd reports SchedulerPodHealthy condition is false (Error, Missing node)]"
```