Kubelet MI exec credential does not propagate Arc HIMDS env vars
- Dominant language
- Go
- Stars
- 14
- Forks
- 28
- Avg merge
- 4d 13h
- Merged PRs (30d)
- 12
Description
Hi team, this is the first of three related observations on the Arc-mode kubelet auth path. They're tightly coupled so I've broken them out into separate issues for clarity and will cross-link them below. Thanks again for v0.1.0 — really enjoying tinkering with it.
## What we observed
When the agent is configured with managed-identity kubelet auth (`node.kubelet.auth.mode: managedidentity`) on a host where the Azure Arc agent (HIMDS) is providing the identity, the exec-credential script the agent writes for kubelet does not appear to propagate the Arc environment variables:
- `IDENTITY_ENDPOINT`
- `IMDS_ENDPOINT`
- `IMDS_API_VERSION`
In our run, kubelet's exec credential then fell back to the default IMDS at `169.254.169.254` (unreachable on a non-Azure host), the Azure Identity client returned an empty token, kubelet sent an empty Bearer to the AKS API server, and the node never joined. (Happy to share repro logs if useful.)
## Repro on the Pi rack
1. Arc-onboard a Raspberry Pi 5 (HIMDS active).
2. Configure agent with `arc.enabled: true` and `node.kubelet.auth.mode: managedidentity`.
3. Start the agent. Kubelet API calls 401 indefinitely; `env` from the exec-credential process shows no `IDENTITY_ENDPOINT`.
## Impact for our scenario
This is the path we expected to use for our edge demo, since Arc was already running on the Pis for the agent's own Azure calls. We worked around it by switching kubelet to `bootstraptoken`, but that path has its own knock-on effects (covered in the two related issues below).
## One possible direction
Two options that came to mind, no strong preference:
1. Have the exec-credential script `source /etc/default/himds` (or whatever Arc places its env in on the host) before invoking the token tool, so the Azure Identity client picks the Arc credential provider.
2. Detect Arc presence (e.g. `systemctl is-active himdsd` or check `/var/opt/azcmagent/`) and write the env vars into the exec stanza of the bootstrap kubeconfig directly.
The second is probably nicer because it survives ordering races on cold boot.
Happy to test a prototype on the Pi rack if helpful.
Version tested: v0.1.0 (git 65d8d38).
---
## Related on this auth path (tightly coupled)
- #186 — `validateExclusiveAuthSettings` treats Arc and bootstrap-token as mutually exclusive
- #187 — Agent could auto-mint kubelet bootstrap-token when Arc/MI/SP credentials are available
---
## Other observations from the same Pi-rack edge demo
- #182 — Arc machine RG and AKS cluster RG must match (hard-coded in v0.1.0)
- #183 — Agent should preflight RBAC permissions before `assignRBACRoles`
- #184 — Hostname casing not normalized; PascalCase machine names fail node validation
- #186 — `validateExclusiveAuthSettings` treats Arc and bootstrap-token as mutually exclusive
- #187 — Agent could auto-mint kubelet bootstrap-token when Arc/MI/SP credentials are available
- #188 — `kube1` systemd-nspawn launcher passes `--network-veth` but actually shares host netns
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.