Cluster create fails on kind stage
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
I tried to create a new cluster, and running with level 9 verbosity, I saw that it looks like it fails when it tries to run kind. What's going on? I ran the exact same procedure a week ago without this error?!
eksctl anywhere create cluster --verbosity=9 --bundles-override eks-anywhere-downloads/bundle-release.yaml -f mgmt_cluster.yaml --force-cleanup 2>&1 | tee /tmp/eksa_create.log
2023-03-31T11:04:18.697+0200 V0 ✅ svc-eksa@vsphere.local user vSphere privileges validated
2023-03-31T11:04:18.697+0200 V0 ✅ Vsphere Provider setup is valid
2023-03-31T11:04:18.760+0200 V0 ✅ Validate certificate for registry mirror
2023-03-31T11:04:18.760+0200 V0 ✅ Validate authentication for git provider
2023-03-31T11:04:18.760+0200 V0 ✅ Create preflight validations pass
2023-03-31T11:04:18.760+0200 V4 Task finished {"task_name": "setup-validate", "duration": "6.142577339s"}
2023-03-31T11:04:18.760+0200 V4 ----------------------------------
2023-03-31T11:04:18.760+0200 V4 Task start {"task_name": "bootstrap-cluster-init"}
2023-03-31T11:04:18.760+0200 V0 Creating new bootstrap cluster
2023-03-31T11:04:18.761+0200 V4 Creating kind cluster {"name": "mgmt-eks-a-cluster", "kubeconfig": "mgmt/generated/mgmt.kind.kubeconfig"}
2023-03-31T11:04:18.761+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1680253451933621095 kind create cluster --name mgmt-eks-a-cluster --kubeconfig mgmt/generated/mgmt.kind.kubeconfig --image harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29 --config mgmt/generated/kind_tmp.yaml"}
2023-03-31T11:08:25.119+0200 V9 docker {"stderr": "Creating cluster \"mgmt-eks-a-cluster\" ...\n • Ensuring node image (harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼 ...\n ✓ Ensuring node image (harbor01.d c.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼\n • Preparing nodes 📦 ...\n ✓ Preparing nodes 📦 \n • Writing configuration 📜 ...\n ✓ Writing configuration 📜\n • Starting control-plane 🕹️ ...\n ✗ Starting control-plane 🕹️\nERROR: failed to create cluster: failed to init node with k ubeadm: command \"docker exec --privileged mgmt-eks-a-cluster-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6\" failed with error: exit status 1\n\nCommand Output: I0331 09:04:22.169296 305 initconfiguration.go:248] loading configuration from \"/kind/kubeadm.conf\"\n[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta3, Kind=JoinConfiguration\nW0331 09:04:22.171261 305 strict.go:55] error unmarshaling configuration schema.GroupVersionKind{Group:\"kubeadm.k8s.io\", Version:\"v1beta3\", Kind:\"ClusterConfiguration\"}: error unmarshaling JSON: while decoding JSON: json: unknown field \"type\"\n[init] Using Kubernetes version: v1.23.16-eks-1-23-16\n[certs] Using certificateDir folder \"/etc/kubernetes/pki\"\nI0331 09:04:22.177230 305 common.go:126] WARNING: tolerating control plane version v1.23.16-eks-1-23-16, assuming that k8s version 1.22.0 is not released yet\nI0331 09:04:22.177418 305 certs.go:112] creating a new certificate authority for ca\n[certs] Generating \"ca\" certificate and key\nI0331 09:04:22.299453 305 certs.go:522] validating certificate period for ca certificate\n[certs] Generating \"apiserver\" certificate and key\n[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost mgmt-eks-a-cluster-control-plane] and IPs [10.96.0.1 172.18.0.2 127.0.0.1]\n[certs] Generating \"apiserver-kubelet-client\" certificate and key\nI0331 09:04:22.756930 305 certs.go:112] creating a new certificate authority for front-proxy-ca\n[certs] Generating \"front-proxy-ca\" certificate and key\nI0331 09:04:22.876813 305 certs.go:522]
[...]
2023-03-31T11:08:25.122+0200 V4 Task finished {"task_name": "bootstrap-cluster-init", "duration": "4m6.362692073s"}
2023-03-31T11:08:25.123+0200 V4 ----------------------------------
2023-03-31T11:08:25.123+0200 V4 Saving checkpoint {"file": "mgmt-checkpoint.yaml"}
2023-03-31T11:08:25.123+0200 V4 Tasks completed {"duration": "4m12.506318354s"}
2023-03-31T11:08:25.123+0200 V3 Logging out from current govc session
2023-03-31T11:08:25.124+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=nwlabvc01-102935-nwlab.se.telenor.net -e GOVC_INSECURE=true -e GOVC_DATACENTER=Rasunda_lab eksa_1680253451933621095 govc session.logout"}
2023-03-31T11:08:25.279+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_DATACENTER=Rasunda_lab -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=nwlabvc01-102935-nwlab.se.telenor.net -e GOVC_INSECURE=true eksa_1680253451933621095 govc session.logout -k"}
2023-03-31T11:08:25.486+0200 V3 Cleaning up long running container {"name": "eksa_1680253451933621095"}
2023-03-31T11:08:25.486+0200 V6 Executing command {"cmd": "/usr/bin/docker rm -f -v eksa_1680253451933621095"}
Error: creating bootstrap cluster: executing create cluster: Creating cluster "mgmt-eks-a-cluster" ...
• Ensuring node image (harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼 ...
✓ Ensuring node image (harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼
• Preparing nodes 📦 ...
✓ Preparing nodes 📦
• Writing configuration 📜 ...
✓ Writing configuration 📜
• Starting control-plane 🕹️ ...
✗ Starting control-plane 🕹️
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged mgmt-eks-a-cluster-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
Command Output: I0331 09:04:22.169296 305 initconfiguration.go:248] loading configuration from "/kind/kubeadm.conf"
[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta3, Kind=JoinConfiguration
W0331 09:04:22.171261 305 strict.go:55] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta3", Kind:"ClusterConfiguration"}: error unmarshaling JSON: while decoding JSON: json: unknown field "type"
[init] Using Kubernetes version: v1.23.16-eks-1-23-16
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0331 09:04:22.177230 305 common.go:126] WARNING: tolerating control plane version v1.23.16-eks-1-23-16, assuming that k8s version 1.22.0 is not released yet
I0331 09:04:22.177418 305 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0331 09:04:22.299453 305 certs.go:522] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost mgmt-eks-a-cluster-control-plane] and IPs [10.96.0.1 172.18.0.2 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0331 09:04:22.756930 305 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I0331 09:04:22.876813 305 certs.go:522] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0331 09:04:22.978872 305 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0331 09:04:23.125195 305 certs.go:522] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost mgmt-eks-a-cluster-control-plane] and IPs [172.18.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
**What you expected to happen**:
The cluster "mgmt" to be created.
**How to reproduce it (as minimally and precisely as possible)**:
/usr/bin/docker exec -i eksa_1680253451933621095 kind create cluster --name mgmt-eks-a-cluster --kubeconfig mgmt/generated/mgmt.kind.kubeconfig --image harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29 --config mgmt/generated/kind_tmp.yaml
reating cluster "mgmt-eks-a-cluster" ...
• Ensuring node image (harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼 ...
✓ Ensuring node image (harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼
• Preparing nodes 📦 ...
✓ Preparing nodes 📦
• Writing configuration 📜 ...
✓ Writing configuration 📜
• Starting control-plane 🕹️ ...
✗ Starting control-plane 🕹️
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged mgmt-eks-a-cluster-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
**Anything else we need to know?**:
**Environment**:
eksctl anywhere version -> v0.14.3
eksctl version -> 0.132.0
Contributor guide
Research direction
Start by reproducing the reported docker exec kind create cluster command with eksctl Anywhere v0.14.3 and inspect mgmt/generated/kind_tmp.yaml alongside the kubeadm error about the unknown type field. Done means the bootstrap cluster is created successfully with the reported vSphere and bundle-override setup, with the failure covered by an appropriate regression check if the repository provides one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100