kubernetes-sigs / kubernetes-sigs/image-builder

Windows: kubelet installed via sc.exe (windows_service_manager=windows_service) ignores KUBELET_KUBEADM_ARGS, breaking Cluster API kubeletExtraArgs

Open
#2,011 0 comments 0 reactions 0 assignees View on GitHub
area/provider/azure kind/bug
Dominant language
Go Template
Stars
550
Forks
508
Avg merge
7d 10h
Merged PRs (30d)
8

Description

## What happened

Windows images built with `windows_service_manager: windows_service` (the sc.exe path added in commit f7d8f02f0) install the kubelet service with a static `binPath` that does **not** consume `/var/lib/kubelet/kubeadm-flags.env`. As a result, every kubelet argument that Cluster API / kubeadm passes via `nodeRegistration.kubeletExtraArgs` is silently dropped at kubelet startup.

The current `images/capi/ansible/windows/roles/kubernetes/tasks/sc.yml` even calls this out in a comment:

> Does not support kubeadm KUBELET_KUBEADM_ARGS which is used by Cluster API to pass extra user args

…but in practice this isn't a workaround that consumers can opt into — it silently breaks any Cluster API–driven Windows workload that relies on `kubeletExtraArgs`. The Linux equivalent (systemd) reads `EnvironmentFile=/var/lib/kubelet/kubeadm-flags.env`, so Linux and Windows nodes built from these images now have divergent semantics for the same Cluster API config.

## Concrete impact (observed)

In the [`cloud-provider-azure-ccm-windows-capz`](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/cloud-provider-azure-ccm-windows-capz/2054230257141026816) job, using `capi-win-2022-containerd:latest` from the CAPZ CI gallery:

- The KubeadmConfig has `kubeletExtraArgs.cloud-provider=external`, `--register-with-taints=node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule`, `--v=2`, `--windows-priorityclass=ABOVE_NORMAL_PRIORITY_CLASS`, etc.
- `kubeadm join` runs cleanly, writes `kubeadm-flags.env`, starts the kubelet service.
- The Windows kubelet service launched via sc.exe ignores all of the above — no `--cloud-provider=external`, no `--register-with-taints`.
- Result: the Windows Node registers with only `node.kubernetes.io/not-ready:NoSchedule`, no `node.cloudprovider.kubernetes.io/uninitialized` taint.
- `cloud-node-manager-windows` logs `Node has no cloud taint, skipping initialization`. `spec.providerID` is never set. Cloud Controller Manager E2E and conformance jobs for Windows fail.

Linux nodes in the same run are unaffected — they get the uninitialized taint (auto-applied by kubelet from `--cloud-provider=external`) and CNM initializes them normally.

A separate Cluster API Provider Azure workaround ([CAPZ #6293](https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/6293)) tried to make the taint explicit via `kubeletExtraArgs.register-with-taints`; it had no effect, because `kubeletExtraArgs` itself is what's being dropped.

## Root cause

`sc.yml` installs the service with:

```yaml
path: >-
"{{ kubernetes_install_path }}\kube-log-runner.exe" --log-file=…\var\log\kubelet\kubelet.log
{{ kubernetes_install_path }}\kubelet.exe --windows-service
--cert-dir=… --config=…\config.yaml --bootstrap-kubeconfig=… --kubeconfig=…
--enable-debugging-handlers --cgroups-per-qos=false --enforce-node-allocatable=""
--container-runtime-endpoint="npipe:////./pipe/containerd-containerd"
--resolv-conf=""
```

`kube-log-runner` is a stdout/stderr redirector; it does not read env files. So `KUBELET_KUBEADM_ARGS` from `/var/lib/kubelet/kubeadm-flags.env` is never appended to the kubelet argv. By contrast, the `nssm.yml` path runs `templates/StartKubelet.ps1`, which does read that file.

## Proposed fix

Make the sc.exe service launch through the same `StartKubelet.ps1` wrapper that the nssm path already uses, so both flows consume `kubeadm-flags.env` identically. Roughly:

```yaml
- name: Render StartKubelet wrapper
ansible.windows.win_template:
src: templates/StartKubelet.ps1
dest: "{{ kubernetes_install_path }}\\StartKubelet.ps1"

- name: Install kubelet as service
ansible.windows.win_service:
name: kubelet
start_mode: auto
path: >-
"{{ kubernetes_install_path }}\kube-log-runner.exe"
--log-file={{ systemdrive.stdout | trim }}/var/log/kubelet/kubelet.log
powershell.exe -ExecutionPolicy Bypass -NonInteractive
-File "{{ kubernetes_install_path }}\StartKubelet.ps1"
```

`kubelet --windows-service` should be dropped from the binPath since kubelet is now a child of the wrapper rather than the service entry point.

#2009 hardens `StartKubelet.ps1` and should land first, so both flows benefit from the same fixes (substring strip of `KUBELET_KUBEADM_ARGS=`, splatting instead of `Invoke-Expression`, optional start-kubelet log).

## Alternatives considered

- Add `--env-file=…` to upstream `kube-log-runner` to splat env-file values onto its child's argv. Cleaner long-term (no PowerShell-as-service), but a Kubernetes-side change with a longer release path.
- Have CAPZ paper over this on the consumer side (e.g., post-join `sc.exe config kubelet binPath= …`). Possible, but pushes a Windows packaging responsibility into every Cluster API provider.

## Environment

- Image: `capi-win-2022-containerd` (Windows Server 2022, containerd) from CAPZ's CI Compute Gallery `ClusterAPI-f72ceb4f-5159-4c26-a0fe-2ea738f0d019`.
- Built with `windows_service_manager: windows_service` (sc.exe path enabled by f7d8f02f0).
- kubelet binary: replaced at boot by CAPZ's `replace-ci-binaries.ps1` with k8s `latest` CI build (currently `v1.37.0-alpha.0.750+17274240a1d95e`). The same bug reproduces with the image's baked kubelet; the version isn't the cause.

/area provider/azure
/kind bug

Contributor guide

Open the contributing guide

Research direction

Start with images/capi/ansible/windows/roles/kubernetes/tasks/sc.yml and compare its service command with templates/StartKubelet.ps1, after reviewing the hardening work in #2009. Verify that the sc.exe path uses the wrapper and consumes kubeadm-flags.env like the nssm path, then check the rendered service command and affected Windows image flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
ansible, kubernetes, powershell
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.