Unable to deploy an EKS Anywhere simple cluster (Bare Metal) - hangs on "Creating new workload" step

Hello everyone!

I'm trying to run a very simple EKS anywhere cluster and it hangs on the creation - "Creating new workload" step

My cluster YAML file:
```
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: bm-anywhere-cluster
spec:
clusterNetwork:
cniConfig:
cilium: {}
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 10.96.0.0/12
controlPlaneConfiguration:
count: 1
endpoint:
host: "192.168.20.199"
machineGroupRef:
kind: TinkerbellMachineConfig
name: bm-anywhere-cluster-cp
datacenterRef:
kind: TinkerbellDatacenterConfig
name: bm-anywhere-cluster
kubernetesVersion: "1.28"
managementCluster:
name: bm-anywhere-cluster
workerNodeGroupConfigurations:
- count: 1
machineGroupRef:
kind: TinkerbellMachineConfig
name: bm-anywhere-cluster
name: md-0

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: TinkerbellDatacenterConfig
metadata:
name: bm-anywhere-cluster
spec:
tinkerbellIP: "192.168.20.200"

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: TinkerbellMachineConfig
metadata:
name: bm-anywhere-cluster-cp
spec:
hardwareSelector:
type: control-plan
osFamily: bottlerocket
templateRef: {}
users:
- name: ec2-user
sshAuthorizedKeys:
- ssh-rsa ssh-rsa

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: TinkerbellMachineConfig
metadata:
name: bm-anywhere-cluster
spec:
hardwareSelector:
type: worker
osFamily: bottlerocket
templateRef: {}
users:
- name: ec2-user
sshAuthorizedKeys:
- ssh-rsa

---
```

Explainmention about the IPs and blocks I provided in the file:
192.168.0.0/16 - there's no existing subnet on my network I just left it as the default
10.96.0.0/12 - there's no existing subnet on my network I just left it as the default

spec.controlPlaneConfiguration.endpoint.host - 192.168.20.199: This is an available IP on my network existing subnet (this subnet has access to the internet).

tinkerbellIP - 192.168.20.200: This is an available IP on my network existing subnet (this subnet has access to the internet)

When I'm creating the EKS Anywhere cluster it hangs.

```
eksctl anywhere create cluster -v=9 -f bm-anywhere-cluster.yaml -z hardware.csv

......
......
......

2024-01-31T12:26:32.523+0200 V0 Creating new workload cluster
2024-01-31T12:26:32.524+0200 V5 Adding extraArgs {"tls-cipher-suites": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"}
2024-01-31T12:26:32.525+0200 V5 Adding extraArgs {"tls-cipher-suites": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"}
2024-01-31T12:26:32.525+0200 V5 Retrier: {"timeout": "2562047h47m16.854775807s", "backoffFactor": null}
2024-01-31T12:26:32.525+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1706696683359269153 kubectl apply -f - --namespace eksa-system --kubeconfig bm-anywhere-cluster/generated/bm-anywhere-cluster.kind.kubeconfig"}
2024-01-31T12:26:32.904+0200 V5 Retry execution successful {"retries": 1, "duration": "378.235373ms"}
2024-01-31T12:26:32.904+0200 V3 Waiting for control plane to be available
2024-01-31T12:26:32.904+0200 V5 Retrier: {"timeout": "1h0m0s", "backoffFactor": null}
2024-01-31T12:26:32.904+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1706696683359269153 kubectl wait --timeout 3600.00s --for=condition=ControlPlaneInitialized clusters.cluster.x-k8s.io/bm-anywhere-cluster --kubeconfig bm-anywhere-cluster/generated/bm-anywhere-cluster.kind.kubeconfig -n eksa-system"}
2024-01-31T13:26:33.096+0200 V9 docker {"stderr": "error: timed out waiting for the condition on clusters/bm-anywhere-cluster\n"}
2024-01-31T13:26:33.096+0200 V5 Error happened during retry {"error": "executing wait: error: timed out waiting for the condition on clusters/bm-anywhere-cluster\n", "retries": 1}
2024-01-31T13:26:33.096+0200 V5 Execution aborted by retry policy
2024-01-31T13:26:33.096+0200 V4 Task finished {"task_name": "workload-cluster-init", "duration": "1h0m0.573792207s"}
2024-01-31T13:26:33.096+0200 V4 ----------------------------------
2024-01-31T13:26:33.096+0200 V4 Task start {"task_name": "collect-cluster-diagnostics"}
2024-01-31T13:26:33.096+0200 V0 collecting cluster diagnostics
2024-01-31T13:26:33.096+0200 V0 collecting management cluster diagnostics
2024-01-31T13:26:33.107+0200 V3 bundle config written {"path": "bm-anywhere-cluster/generated/bootstrap-cluster-2024-01-31T13:26:33+02:00-bundle.yaml"}
2024-01-31T13:26:33.107+0200 V1 creating temporary namespace for diagnostic collector {"namespace": "eksa-diagnostics"}
2024-01-31T13:26:33.107+0200 V5 Retrier: {"timeout": "2562047h47m16.854775807s", "backoffFactor": null}
2024-01-31T13:26:33.107+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1706696683359269153 kubectl create namespace eksa-diagnostics --kubeconfig bm-anywhere-cluster/generated/bm-anywhere-cluster.kind.kubeconfig"}
2024-01-31T13:26:33.263+0200 V5 Retry execution successful {"retries": 1, "duration": "155.703042ms"}
2024-01-31T13:26:33.263+0200 V1 creating temporary ClusterRole and RoleBinding for diagnostic collector
2024-01-31T13:26:33.263+0200 V5 Retrier: {"timeout": "2562047h47m16.854775807s", "backoffFactor": null}
2024-01-31T13:26:33.263+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1706696683359269153 kubectl apply -f - --kubeconfig bm-anywhere-cluster/generated/bm-anywhere-cluster.kind.kubeconfig"}
2024-01-31T13:26:33.510+0200 V5 Retry execution successful {"retries": 1, "duration": "246.957482ms"}
2024-01-31T13:26:33.510+0200 V0 ⏳ Collecting support bundle from cluster, this can take a while {"cluster": "bootstrap-cluster", "bundle": "bm-anywhere-cluster/generated/bootstrap-cluster-2024-01-31T13:26:33+02:00-bundle.yaml", "since": "2024-01-31T10:26:33.107+0200", "kubeconfig": "bm-anywhere-cluster/generated/bm-anywhere-cluster.kind.kubeconfig"}
2024-01-31T13:26:33.510+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1706696683359269153 support-bundle bm-anywhere-cluster/generated/bootstrap-cluster-2024-01-31T13:26:33+02:00-bundle.yaml --kubeconfig bm-anywhere-cluster/generated/bm-anywhere-cluster.kind.kubeconfig --interactive=false --since-time 2024-01-31T10:26:33.107500907+02:00"}
2024-01-31T13:26:40.926+0200 V0 Support bundle archive created {"path": "support-bundle-2024-01-31T11_26_33.tar.gz"}
2024-01-31T13:26:40.926+0200 V0 Analyzing support bundle {"bundle": "bm-anywhere-cluster/generated/bootstrap-cluster-2024-01-31T13:26:33+02:00-bundle.yaml", "archive": "support-bundle-2024-01-31T11_26_33.tar.gz"}
2024-01-31T13:26:40.926+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1706696683359269153 support-bundle analyze bm-anywhere-cluster/generated/bootstrap-cluster-2024-01-31T13:26:33+02:00-bundle.yaml --bundle support-bundle-2024-01-31T11_26_33.tar.gz --output json"}
2024-01-31T13:26:41.211+0200 V0 Analysis output generated {"path": "bm-anywhere-cluster/generated/bootstrap-cluster-2024-01-31T13:26:41+02:00-analysis.yaml"}
2024-01-31T13:26:41.211+0200 V1 cleaning up temporary roles for diagnostic collectors
2024-01-31T13:26:41.211+0200 V5 Retrier: {"timeout": "2562047h47m16.854775807s", "backoffFactor": null}
2024-01-31T13:26:41.211+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1706696683359269153 kubectl delete -f - --kubeconfig bm-anywhere-cluster/generated/bm-anywhere-cluster.kind.kubeconfig"}
2024-01-31T13:26:41.374+0200 V5 Retry execution successful {"retries": 1, "duration": "163.248996ms"}
2024-01-31T13:26:41.374+0200 V1 cleaning up temporary namespace for diagnostic collectors {"namespace": "eksa-diagnostics"}
2024-01-31T13:26:41.374+0200 V5 Retrier: {"timeout": "2562047h47m16.854775807s", "backoffFactor": null}
2024-01-31T13:26:41.374+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1706696683359269153 kubectl delete namespace eksa-diagnostics --kubeconfig bm-anywhere-cluster/generated/bm-anywhere-cluster.kind.kubeconfig"}
2024-01-31T13:26:46.905+0200 V5 Retry execution successful {"retries": 1, "duration": "5.530672342s"}
2024-01-31T13:26:46.905+0200 V0 collecting workload cluster diagnostics
2024-01-31T13:26:46.905+0200 V4 Task finished {"task_name": "collect-cluster-diagnostics", "duration": "13.808803162s"}
2024-01-31T13:26:46.905+0200 V4 ----------------------------------
2024-01-31T13:26:46.905+0200 V4 Saving checkpoint {"file": "bm-anywhere-cluster-checkpoint.yaml"}
2024-01-31T13:26:46.906+0200 V4 Tasks completed {"duration": "1h2m1.226833399s"}
2024-01-31T13:26:46.906+0200 V3 Cleaning up long running container {"name": "eksa_1706696683359269153"}
2024-01-31T13:26:46.906+0200 V6 Executing command {"cmd": "/usr/bin/docker rm -f -v eksa_1706696683359269153"}
Error: waiting for control plane to be ready: executing wait: executing wait: error: timed out waiting for the condition on clusters/bm-anywhere-cluster
```

Docker boots logs shows:
```
docker logs -f boots
{"level":"info","ts":1706696738.415058,"caller":"boots/main.go:119","msg":"starting","service":"github.com/tinkerbell/boots","pkg":"main","version":"8fd5c38"}
{"level":"info","ts":1706696738.4281049,"caller":"boots/main.go:186","msg":"serving iPXE binaries from local HTTP server","service":"github.com/tinkerbell/boots","pkg":"main","addr":"192.168.20.116/ipxe/"}
{"level":"info","ts":1706696738.4281282,"caller":"boots/main.go:128","msg":"serving syslog","service":"github.com/tinkerbell/boots","pkg":"main","addr":"192.168.20.116:514"}
{"level":"info","ts":1706696738.4281378,"caller":"boots/main.go:205","msg":"serving dhcp","service":"github.com/tinkerbell/boots","pkg":"main","addr":"0.0.0.0:67"}
{"level":"info","ts":1706696738.4281623,"caller":"boots/main.go:212","msg":"serving http","service":"github.com/tinkerbell/boots","pkg":"main","addr":"192.168.20.116:80"}
{"level":"info","ts":1706696738.4282918,"logger":"github.com/tinkerbell/ipxedust","caller":"ipxedust@v0.0.0-20230118215055-b00d1b371ddf/ipxedust.go:194","msg":"serving iPXE binaries via TFTP","service":"github.com/tinkerbell/boots","addr":"0.0.0.0:69","timeout":5,"singlePortEnabled":true}
{"level":"info","ts":1706699115.5887766,"caller":"dhcp4-go@v0.0.0-20190402165401-39c137f31ad3/handler.go:105","msg":"","service":"github.com/tinkerbell/boots","pkg":"dhcp","pkg":"dhcp","event":"recv","mac":"00:50:b6:ec:9a:53","via":"0.0.0.0","iface":"eno2","xid":"\"84:13:be:6b\"","type":"DHCPREQUEST"}
{"level":"info","ts":1706699115.5889626,"caller":"boots/dhcp.go:88","msg":"parsed option82/circuitid","service":"github.com/tinkerbell/boots","pkg":"main","mac":"00:50:b6:ec:9a:53","circuitID":""}
{"level":"error","ts":1706699115.5893145,"caller":"boots/dhcp.go:101","msg":"retrieved job is empty","service":"github.com/tinkerbell/boots","pkg":"main","type":"DHCPREQUEST","mac":"00:50:b6:ec:9a:53","error":"discover from dhcp message: no hardware found","errorVerbose":"no hardware found\ngithub.com/tinkerbell/boots/client/kubernetes.(*Finder).ByMAC\n\tgithub.com/tinkerbell/boots/client/kubernetes/hardware_finder.go:96\ngithub.com/tinkerbell/boots/job.(*Creator).CreateFromDHCP\n\tgithub.com/tinkerbell/boots/job/job.go:107\nmain.dhcpHandler.serve\n\tgithub.com/tinkerbell/boots/cmd/boots/dhcp.go:99\nmain.dhcpHandler.ServeDHCP.func1\n\tgithub.com/tinkerbell/boots/cmd/boots/dhcp.go:60\ngithub.com/gammazero/workerpool.(*WorkerPool).dispatch.func1\n\tgithub.com/gammazero/workerpool@v0.0.0-20200311205957-7b00833861c6/workerpool.go:169\nruntime.goexit\n\truntime/asm_amd64.s:1571\ndiscover from dhcp message"}
```

Start by reviewing the generated support bundle and the boots logs, then trace the DHCP path in boots/dhcp.go through boots/client/kubernetes/hardware_finder.go. Reproduce with the supplied cluster YAML and hardware.csv; done means the control plane reaches ControlPlaneInitialized and the provisioning failure is explained or fixed.

Written by the indexing model from the issue text.