clusterNetwork.dns.resolvConf does not work as expected
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
I did test vSphere cluster creation with `clusterNetwork.dns.resolvConf.path = /etc/test-resolv.conf` , on both Ubuntu and BottleRocket OS.
For BR:
The `--resolv-conf` arg did show up in KCP kubelet extra arg as expected, but it was not passed to the CP node when running kubelet. Thus kubelet was using the default `/etc/resolv.conf` and continued the provision.
```sh
$ ps -ef | grep kubelet
...
kubelet --cloud-provider external --kubeconfig /etc/kubernetes/kubelet/kubeconfig --bootstrap-kubeconfig /etc/kubernetes/kubelet/bootstrap-kubeconfig --config /etc/kubernetes/kubelet/config --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --containerd=/run/containerd/containerd.sock --root-dir /var/lib/kubelet --cert-dir /var/lib/kubelet/pki --node-ip 195.17.37.241 --node-labels --register-with-taints --pod-infra-container-image public.ecr.aws/eks-distro/kubernetes/pause:v1.26.4-eks-1-26-9
```
```sh
$ kubectl get --raw "/api/v1/nodes/joeywang-cjqf5/proxy/configz" | jq
{
"kubeletconfig": {
...
"resolvConf": "/etc/resolv.conf",
...
}
}
```
However, Ubuntu test runs successfully where the custom path got picked by `kubelet` command and was applied to current config.
```sh
$ ps -ef | grep kubelet
...
/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --anonymous-auth=false --cloud-provider=external --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=public.ecr.aws/eks-distro/kubernetes/pause:3.9 --read-only-port=0 --resolv-conf=/etc/test-resolv.conf --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
```
```sh
$ kubectl get --raw "/api/v1/nodes/joeywang-ubuntu-gw984/proxy/configz" | jq
{
"kubeletconfig": {
...
"resolvConf": "/etc/test-resolv.conf",
}
}
```
Although Ubuntu test did pick up the custom DNS, both kubelet logs show a deprecation error:
```sh
Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
```
Nutanix Ubuntu test shows similar behavior as vSphere Ubuntu test.
```sh
$ ps -ef | grep kubelet
...
root 1786 1 2 20:04 ? 00:00:11 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --eviction-hard=nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0% --pod-infra-container-image=public.ecr.aws/eks-distro/kubernetes/pause:3.9 --resolv-conf=/etc/test-resolv.conf --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
```
```sh
$ kubectl get --raw "/api/v1/nodes/joeywang-nutanix-nqvwz/proxy/configz" | jq
{
"kubeletconfig": {
...
"resolvConf": "/etc/test-resolv.conf",
}
}
```
**What you expected to happen**:
BottleRocket OS should respect the custom DNS option in EKS-A cluster spec, similar to Ubuntu.
We also need to use kubelet configuration file instead of the deprecated args passed to kubelet.
**How to reproduce it (as minimally and precisely as possible)**:
**Anything else we need to know?**:
**Environment**:
- EKS Anywhere Release:
- EKS Distro Release:
Contributor guide
Research direction
Compare how the cluster's kubelet settings are applied on Bottlerocket and Ubuntu, starting with the kubelet extra args and the kubelet configuration file mentioned in the report. Verify the change on a Bottlerocket vSphere cluster using a custom resolvConf path; done means kubelet uses that path as Ubuntu does, without relying on the deprecated --resolv-conf argument.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100