Azure / Azure/AKSFlexNode

validateExclusiveAuthSettings treats Arc and bootstrap-token as mutually exclusive

Open
#186 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14
Forks
28
Avg merge
4d 13h
Merged PRs (30d)
12

Description

Hi team, second of three related observations on the Arc-mode kubelet auth path. Cross-links to the others at the bottom.

## What we observed

`validateExclusiveAuthSettings` in `pkg/config/validate.go` treats `arc.enabled` and `bootstrapToken` as mutually exclusive. Picking either appears to disable the other across the whole agent.

In our scenario these two felt like they were doing different jobs:

- `arc.enabled` controls the agent's own Azure machine identity (used for `enrich-cluster-config`, `assignRBACRoles`, AKS RP calls, etc).
- `bootstrapToken` controls how kubelet authenticates to the API server.

When we picked `bootstrapToken` to work around the MI exec issue (see related issue below), Arc was also disabled, which disabled `enrich-cluster-config`. The agent then no longer refreshed the cluster CA on its own, and when the API-server CA rotated (this happened after we enabled AAD on the cluster), the bootstrap kubeconfig went stale and kubelet bootstrap failed until we manually re-fetched the CA.

## Impact for our scenario

The Arc-mode MI happy path doesn't yet work end-to-end for us (separate issue), so the only path that actually completes is "bootstrap-token for kubelet". But that path also disables Arc for the agent, which gives up `enrich-cluster-config` and leaves us with a stale CA on rotation. Feels like we're being forced to pick one cost or the other.

## One possible direction

Would it be reasonable to treat these two as orthogonal? Roughly:

| `arc.enabled` | `bootstrapToken` | `node.kubelet.auth.mode` | Resulting behavior |
|---|---|---|---|
| true | unset | mi | Arc-driven everything; kubelet uses MI exec |
| true | set | bootstraptoken | Arc for the agent's Azure-side ops (incl. `enrich-cluster-config`), bootstrap-token brings kubelet up |
| false | set | bootstraptoken | "Bring-your-own-everything" path (existing VM demos) |

The middle row would unblock us today without depending on the MI exec fix landing first. Totally happy to be told we're holding it wrong if there's a reason these are exclusive that we missed.

Version tested: v0.1.0 (git 65d8d38).

---

## Related on this auth path (tightly coupled)

- #185 — Kubelet MI exec credential does not propagate Arc HIMDS env vars
- #187 — Agent could auto-mint kubelet bootstrap-token when Arc/MI/SP credentials are available

---

## Other observations from the same Pi-rack edge demo

- #182 — Arc machine RG and AKS cluster RG must match (hard-coded in v0.1.0)
- #183 — Agent should preflight RBAC permissions before `assignRBACRoles`
- #184 — Hostname casing not normalized; PascalCase machine names fail node validation
- #185 — Kubelet MI exec credential does not propagate Arc HIMDS env vars
- #187 — Agent could auto-mint kubelet bootstrap-token when Arc/MI/SP credentials are available
- #188 — `kube1` systemd-nspawn launcher passes `--network-veth` but actually shares host netns

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in pkg/config/validate.go at validateExclusiveAuthSettings and trace how arc.enabled, bootstrapToken, and node.kubelet.auth.mode are validated and applied. Compare the current behavior with the three configuration combinations in the issue, then verify that Arc agent operations and bootstrap-token kubelet authentication can coexist without disabling either path.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
authentication
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.