apache / apache/cloudstack

CKS cluster doesn't come up if "cloud.kubernetes.cluster.experimental.features.enabled" setting is enabled

Open
#11,596 2 comments 0 reactions 0 assignees View on GitHub
component:cks type:bug
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

### problem

CKS cluster doesn't come up if "cloud.kubernetes.cluster.experimental.features.enabled" setting is enabled

### versions

ACS 4.20.1, 4.21

### The steps to reproduce the bug

1. Enable the global setting "cloud.kubernetes.cluster.experimental.features.enabled" setting

2. Launch a cks cluster, enable private registry and provide your docker hub credentials

Image

dockerregistryurl= "https://index.docker.io/v1/"
dockerregistryusername= <>
dockerregistrypassword=<>

3. CKS cluster is struck in starting state

4. On logging into control node found the following errors

```
root@example-cluster-control-19927ba5b6e:/opt/bin# crictl images
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/opt/bin/crictl.yaml"
WARN[0000] Image connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///run/crio/crio.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/crio/crio.sock: connect: no such file or directory"
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: no such file or directory"
FATA[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: no such file or directory"

```

kublet logs

```

Sep 08 05:18:03 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 08 05:18:03 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 08 05:18:13 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 41.
Sep 08 05:18:13 example-cluster-control-19927ba5b6e systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:13 example-cluster-control-19927ba5b6e systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:14 example-cluster-control-19927ba5b6e kubelet[31660]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's -->
Sep 08 05:18:14 example-cluster-control-19927ba5b6e kubelet[31660]: E0908 05:18:14.034670 31660 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubele>
Sep 08 05:18:14 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 08 05:18:14 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 42.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e kubelet[32393]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's -->
Sep 08 05:18:24 example-cluster-control-19927ba5b6e kubelet[32393]: E0908 05:18:24.237711 32393 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubele>

```

The file

"/etc/containerd/config.toml" is not populated correctly

### What to do about it?

Cloudstack should support private container registry

Contributor guide

Open the contributing guide

Research direction

Start with the CKS cluster provisioning path and the generated /etc/containerd/config.toml, using the kubelet and crictl errors as failure evidence. Reproduce with the experimental-features setting and private Docker Hub registry credentials; done means the cluster leaves starting state and kubelet and containerd initialize correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.