aws / aws/eks-anywhere

clusterNetwork.dns.resolvConf does not work as expected

Open
#5,879 0 comments 0 reactions 0 assignees View on GitHub
area/cli kind/bug team/ce
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:

I did test vSphere cluster creation with `clusterNetwork.dns.resolvConf.path = /etc/test-resolv.conf` , on both Ubuntu and BottleRocket OS.

For BR:
The `--resolv-conf` arg did show up in KCP kubelet extra arg as expected, but it was not passed to the CP node when running kubelet. Thus kubelet was using the default `/etc/resolv.conf` and continued the provision.

```sh
$ ps -ef | grep kubelet

...
kubelet --cloud-provider external --kubeconfig /etc/kubernetes/kubelet/kubeconfig --bootstrap-kubeconfig /etc/kubernetes/kubelet/bootstrap-kubeconfig --config /etc/kubernetes/kubelet/config --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --containerd=/run/containerd/containerd.sock --root-dir /var/lib/kubelet --cert-dir /var/lib/kubelet/pki --node-ip 195.17.37.241 --node-labels --register-with-taints --pod-infra-container-image public.ecr.aws/eks-distro/kubernetes/pause:v1.26.4-eks-1-26-9
```

```sh
$ kubectl get --raw "/api/v1/nodes/joeywang-cjqf5/proxy/configz" | jq

{
"kubeletconfig": {
...
"resolvConf": "/etc/resolv.conf",
...
}
}
```

However, Ubuntu test runs successfully where the custom path got picked by `kubelet` command and was applied to current config.

```sh
$ ps -ef | grep kubelet

...
/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --anonymous-auth=false --cloud-provider=external --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=public.ecr.aws/eks-distro/kubernetes/pause:3.9 --read-only-port=0 --resolv-conf=/etc/test-resolv.conf --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
```

```sh
$ kubectl get --raw "/api/v1/nodes/joeywang-ubuntu-gw984/proxy/configz" | jq
{
"kubeletconfig": {
...
"resolvConf": "/etc/test-resolv.conf",
}
}
```
Although Ubuntu test did pick up the custom DNS, both kubelet logs show a deprecation error:

```sh
Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
```

Nutanix Ubuntu test shows similar behavior as vSphere Ubuntu test.

```sh
$ ps -ef | grep kubelet

...
root 1786 1 2 20:04 ? 00:00:11 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --eviction-hard=nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0% --pod-infra-container-image=public.ecr.aws/eks-distro/kubernetes/pause:3.9 --resolv-conf=/etc/test-resolv.conf --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
```

```sh
$ kubectl get --raw "/api/v1/nodes/joeywang-nutanix-nqvwz/proxy/configz" | jq

{
"kubeletconfig": {
...
"resolvConf": "/etc/test-resolv.conf",
}
}
```

**What you expected to happen**:

BottleRocket OS should respect the custom DNS option in EKS-A cluster spec, similar to Ubuntu.

We also need to use kubelet configuration file instead of the deprecated args passed to kubelet.

**How to reproduce it (as minimally and precisely as possible)**:

**Anything else we need to know?**:

**Environment**:
- EKS Anywhere Release:
- EKS Distro Release:

Contributor guide

Open the contributing guide

Research direction

Compare how the cluster's kubelet settings are applied on Bottlerocket and Ubuntu, starting with the kubelet extra args and the kubelet configuration file mentioned in the report. Verify the change on a Bottlerocket vSphere cluster using a custom resolvConf path; done means kubelet uses that path as Ubuntu does, without relying on the deprecated --resolv-conf argument.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.