aws / aws/eks-anywhere

Cluster create fails on kind stage

Open
#5,475 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:

I tried to create a new cluster, and running with level 9 verbosity, I saw that it looks like it fails when it tries to run kind. What's going on? I ran the exact same procedure a week ago without this error?!

eksctl anywhere create cluster --verbosity=9 --bundles-override eks-anywhere-downloads/bundle-release.yaml -f mgmt_cluster.yaml --force-cleanup 2>&1 | tee /tmp/eksa_create.log

2023-03-31T11:04:18.697+0200 V0 ✅ svc-eksa@vsphere.local user vSphere privileges validated
2023-03-31T11:04:18.697+0200 V0 ✅ Vsphere Provider setup is valid
2023-03-31T11:04:18.760+0200 V0 ✅ Validate certificate for registry mirror
2023-03-31T11:04:18.760+0200 V0 ✅ Validate authentication for git provider
2023-03-31T11:04:18.760+0200 V0 ✅ Create preflight validations pass
2023-03-31T11:04:18.760+0200 V4 Task finished {"task_name": "setup-validate", "duration": "6.142577339s"}
2023-03-31T11:04:18.760+0200 V4 ----------------------------------
2023-03-31T11:04:18.760+0200 V4 Task start {"task_name": "bootstrap-cluster-init"}
2023-03-31T11:04:18.760+0200 V0 Creating new bootstrap cluster
2023-03-31T11:04:18.761+0200 V4 Creating kind cluster {"name": "mgmt-eks-a-cluster", "kubeconfig": "mgmt/generated/mgmt.kind.kubeconfig"}
2023-03-31T11:04:18.761+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1680253451933621095 kind create cluster --name mgmt-eks-a-cluster --kubeconfig mgmt/generated/mgmt.kind.kubeconfig --image harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29 --config mgmt/generated/kind_tmp.yaml"}

2023-03-31T11:08:25.119+0200 V9 docker {"stderr": "Creating cluster \"mgmt-eks-a-cluster\" ...\n • Ensuring node image (harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼 ...\n ✓ Ensuring node image (harbor01.d c.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼\n • Preparing nodes 📦 ...\n ✓ Preparing nodes 📦 \n • Writing configuration 📜 ...\n ✓ Writing configuration 📜\n • Starting control-plane 🕹️ ...\n ✗ Starting control-plane 🕹️\nERROR: failed to create cluster: failed to init node with k ubeadm: command \"docker exec --privileged mgmt-eks-a-cluster-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6\" failed with error: exit status 1\n\nCommand Output: I0331 09:04:22.169296 305 initconfiguration.go:248] loading configuration from \"/kind/kubeadm.conf\"\n[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta3, Kind=JoinConfiguration\nW0331 09:04:22.171261 305 strict.go:55] error unmarshaling configuration schema.GroupVersionKind{Group:\"kubeadm.k8s.io\", Version:\"v1beta3\", Kind:\"ClusterConfiguration\"}: error unmarshaling JSON: while decoding JSON: json: unknown field \"type\"\n[init] Using Kubernetes version: v1.23.16-eks-1-23-16\n[certs] Using certificateDir folder \"/etc/kubernetes/pki\"\nI0331 09:04:22.177230 305 common.go:126] WARNING: tolerating control plane version v1.23.16-eks-1-23-16, assuming that k8s version 1.22.0 is not released yet\nI0331 09:04:22.177418 305 certs.go:112] creating a new certificate authority for ca\n[certs] Generating \"ca\" certificate and key\nI0331 09:04:22.299453 305 certs.go:522] validating certificate period for ca certificate\n[certs] Generating \"apiserver\" certificate and key\n[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost mgmt-eks-a-cluster-control-plane] and IPs [10.96.0.1 172.18.0.2 127.0.0.1]\n[certs] Generating \"apiserver-kubelet-client\" certificate and key\nI0331 09:04:22.756930 305 certs.go:112] creating a new certificate authority for front-proxy-ca\n[certs] Generating \"front-proxy-ca\" certificate and key\nI0331 09:04:22.876813 305 certs.go:522]
[...]
2023-03-31T11:08:25.122+0200 V4 Task finished {"task_name": "bootstrap-cluster-init", "duration": "4m6.362692073s"}
2023-03-31T11:08:25.123+0200 V4 ----------------------------------
2023-03-31T11:08:25.123+0200 V4 Saving checkpoint {"file": "mgmt-checkpoint.yaml"}
2023-03-31T11:08:25.123+0200 V4 Tasks completed {"duration": "4m12.506318354s"}
2023-03-31T11:08:25.123+0200 V3 Logging out from current govc session
2023-03-31T11:08:25.124+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=nwlabvc01-102935-nwlab.se.telenor.net -e GOVC_INSECURE=true -e GOVC_DATACENTER=Rasunda_lab eksa_1680253451933621095 govc session.logout"}
2023-03-31T11:08:25.279+0200 V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_DATACENTER=Rasunda_lab -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=nwlabvc01-102935-nwlab.se.telenor.net -e GOVC_INSECURE=true eksa_1680253451933621095 govc session.logout -k"}
2023-03-31T11:08:25.486+0200 V3 Cleaning up long running container {"name": "eksa_1680253451933621095"}
2023-03-31T11:08:25.486+0200 V6 Executing command {"cmd": "/usr/bin/docker rm -f -v eksa_1680253451933621095"}
Error: creating bootstrap cluster: executing create cluster: Creating cluster "mgmt-eks-a-cluster" ...
• Ensuring node image (harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼 ...
✓ Ensuring node image (harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼
• Preparing nodes 📦 ...
✓ Preparing nodes 📦
• Writing configuration 📜 ...
✓ Writing configuration 📜
• Starting control-plane 🕹️ ...
✗ Starting control-plane 🕹️
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged mgmt-eks-a-cluster-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1

Command Output: I0331 09:04:22.169296 305 initconfiguration.go:248] loading configuration from "/kind/kubeadm.conf"
[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta3, Kind=JoinConfiguration
W0331 09:04:22.171261 305 strict.go:55] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta3", Kind:"ClusterConfiguration"}: error unmarshaling JSON: while decoding JSON: json: unknown field "type"
[init] Using Kubernetes version: v1.23.16-eks-1-23-16
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0331 09:04:22.177230 305 common.go:126] WARNING: tolerating control plane version v1.23.16-eks-1-23-16, assuming that k8s version 1.22.0 is not released yet
I0331 09:04:22.177418 305 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0331 09:04:22.299453 305 certs.go:522] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost mgmt-eks-a-cluster-control-plane] and IPs [10.96.0.1 172.18.0.2 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0331 09:04:22.756930 305 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I0331 09:04:22.876813 305 certs.go:522] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0331 09:04:22.978872 305 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0331 09:04:23.125195 305 certs.go:522] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost mgmt-eks-a-cluster-control-plane] and IPs [172.18.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key

**What you expected to happen**:
The cluster "mgmt" to be created.

**How to reproduce it (as minimally and precisely as possible)**:
/usr/bin/docker exec -i eksa_1680253451933621095 kind create cluster --name mgmt-eks-a-cluster --kubeconfig mgmt/generated/mgmt.kind.kubeconfig --image harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29 --config mgmt/generated/kind_tmp.yaml
reating cluster "mgmt-eks-a-cluster" ...
• Ensuring node image (harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼 ...
✓ Ensuring node image (harbor01.dc.tnse.se/eks-anywhere/kubernetes-sigs/kind/node:v1.23.16-eks-d-1-23-16-eks-a-29) 🖼
• Preparing nodes 📦 ...
✓ Preparing nodes 📦
• Writing configuration 📜 ...
✓ Writing configuration 📜
• Starting control-plane 🕹️ ...
✗ Starting control-plane 🕹️
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged mgmt-eks-a-cluster-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1

**Anything else we need to know?**:

**Environment**:
eksctl anywhere version -> v0.14.3
eksctl version -> 0.132.0

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported docker exec kind create cluster command with eksctl Anywhere v0.14.3 and inspect mgmt/generated/kind_tmp.yaml alongside the kubeadm error about the unknown type field. Done means the bootstrap cluster is created successfully with the reported vSphere and bundle-override setup, with the failure covered by an appropriate regression check if the repository provides one.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.