kubernetes-sigs / kubernetes-sigs/image-builder
Windows: kubelet installed via sc.exe (windows_service_manager=windows_service) ignores KUBELET_KUBEADM_ARGS, breaking Cluster API kubeletExtraArgs
- Dominant language
- Go Template
- Stars
- 550
- Forks
- 508
- Avg merge
- 7d 10h
- Merged PRs (30d)
- 8
Description
## What happened
Windows images built with `windows_service_manager: windows_service` (the sc.exe path added in commit f7d8f02f0) install the kubelet service with a static `binPath` that does **not** consume `/var/lib/kubelet/kubeadm-flags.env`. As a result, every kubelet argument that Cluster API / kubeadm passes via `nodeRegistration.kubeletExtraArgs` is silently dropped at kubelet startup.
The current `images/capi/ansible/windows/roles/kubernetes/tasks/sc.yml` even calls this out in a comment:
> Does not support kubeadm KUBELET_KUBEADM_ARGS which is used by Cluster API to pass extra user args
…but in practice this isn't a workaround that consumers can opt into — it silently breaks any Cluster API–driven Windows workload that relies on `kubeletExtraArgs`. The Linux equivalent (systemd) reads `EnvironmentFile=/var/lib/kubelet/kubeadm-flags.env`, so Linux and Windows nodes built from these images now have divergent semantics for the same Cluster API config.
## Concrete impact (observed)
In the [`cloud-provider-azure-ccm-windows-capz`](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/cloud-provider-azure-ccm-windows-capz/2054230257141026816) job, using `capi-win-2022-containerd:latest` from the CAPZ CI gallery:
- The KubeadmConfig has `kubeletExtraArgs.cloud-provider=external`, `--register-with-taints=node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule`, `--v=2`, `--windows-priorityclass=ABOVE_NORMAL_PRIORITY_CLASS`, etc.
- `kubeadm join` runs cleanly, writes `kubeadm-flags.env`, starts the kubelet service.
- The Windows kubelet service launched via sc.exe ignores all of the above — no `--cloud-provider=external`, no `--register-with-taints`.
- Result: the Windows Node registers with only `node.kubernetes.io/not-ready:NoSchedule`, no `node.cloudprovider.kubernetes.io/uninitialized` taint.
- `cloud-node-manager-windows` logs `Node has no cloud taint, skipping initialization`. `spec.providerID` is never set. Cloud Controller Manager E2E and conformance jobs for Windows fail.
Linux nodes in the same run are unaffected — they get the uninitialized taint (auto-applied by kubelet from `--cloud-provider=external`) and CNM initializes them normally.
A separate Cluster API Provider Azure workaround ([CAPZ #6293](https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/6293)) tried to make the taint explicit via `kubeletExtraArgs.register-with-taints`; it had no effect, because `kubeletExtraArgs` itself is what's being dropped.
## Root cause
`sc.yml` installs the service with:
```yaml
path: >-
"{{ kubernetes_install_path }}\kube-log-runner.exe" --log-file=…\var\log\kubelet\kubelet.log
{{ kubernetes_install_path }}\kubelet.exe --windows-service
--cert-dir=… --config=…\config.yaml --bootstrap-kubeconfig=… --kubeconfig=…
--enable-debugging-handlers --cgroups-per-qos=false --enforce-node-allocatable=""
--container-runtime-endpoint="npipe:////./pipe/containerd-containerd"
--resolv-conf=""
```
`kube-log-runner` is a stdout/stderr redirector; it does not read env files. So `KUBELET_KUBEADM_ARGS` from `/var/lib/kubelet/kubeadm-flags.env` is never appended to the kubelet argv. By contrast, the `nssm.yml` path runs `templates/StartKubelet.ps1`, which does read that file.
## Proposed fix
Make the sc.exe service launch through the same `StartKubelet.ps1` wrapper that the nssm path already uses, so both flows consume `kubeadm-flags.env` identically. Roughly:
```yaml
- name: Render StartKubelet wrapper
ansible.windows.win_template:
src: templates/StartKubelet.ps1
dest: "{{ kubernetes_install_path }}\\StartKubelet.ps1"
- name: Install kubelet as service
ansible.windows.win_service:
name: kubelet
start_mode: auto
path: >-
"{{ kubernetes_install_path }}\kube-log-runner.exe"
--log-file={{ systemdrive.stdout | trim }}/var/log/kubelet/kubelet.log
powershell.exe -ExecutionPolicy Bypass -NonInteractive
-File "{{ kubernetes_install_path }}\StartKubelet.ps1"
```
`kubelet --windows-service` should be dropped from the binPath since kubelet is now a child of the wrapper rather than the service entry point.
#2009 hardens `StartKubelet.ps1` and should land first, so both flows benefit from the same fixes (substring strip of `KUBELET_KUBEADM_ARGS=`, splatting instead of `Invoke-Expression`, optional start-kubelet log).
## Alternatives considered
- Add `--env-file=…` to upstream `kube-log-runner` to splat env-file values onto its child's argv. Cleaner long-term (no PowerShell-as-service), but a Kubernetes-side change with a longer release path.
- Have CAPZ paper over this on the consumer side (e.g., post-join `sc.exe config kubelet binPath= …`). Possible, but pushes a Windows packaging responsibility into every Cluster API provider.
## Environment
- Image: `capi-win-2022-containerd` (Windows Server 2022, containerd) from CAPZ's CI Compute Gallery `ClusterAPI-f72ceb4f-5159-4c26-a0fe-2ea738f0d019`.
- Built with `windows_service_manager: windows_service` (sc.exe path enabled by f7d8f02f0).
- kubelet binary: replaced at boot by CAPZ's `replace-ci-binaries.ps1` with k8s `latest` CI build (currently `v1.37.0-alpha.0.750+17274240a1d95e`). The same bug reproduces with the image's baked kubelet; the version isn't the cause.
/area provider/azure
/kind bug
Contributor guide
Research direction
Start with images/capi/ansible/windows/roles/kubernetes/tasks/sc.yml and compare its service command with templates/StartKubelet.ps1, after reviewing the hardening work in #2009. Verify that the sc.exe path uses the wrapper and consumes kubeadm-flags.env like the nssm path, then check the rendered service command and affected Windows image flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ansible, kubernetes, powershell
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100