unbounded-stable: Site/cluster node CIDR does not match any node, and boulderlab blocks the release gate
- Dominant language
- Go
- Stars
- 28
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 55
Description
## Summary
`unbounded-stable` has not completed a soak since 2026-08-12. Two separate
problems, both currently blocking releases from deploying there.
## 1. `Site/cluster` claims a node CIDR that matches nothing
```
Site/cluster spec.nodeCidrs: ["10.200.0.0/16"]
AKS nodes: 10.224.0.4 .. 10.224.0.11
```
Disjoint ranges. Every AKS node carries **no** `unbounded-cloud.io/*` label and
belongs to no Site, because assignment matches InternalIP against
`spec.nodeCidrs` (`internal/net/controller/site_controller.go:2534`). Nodes that
belong to no Site get no pod CIDR assignment from `computePodCIDRsForNode`.
The other Sites are consistent: `stable` is `10.1.0.0/16` with its node at
`10.1.0.10`, `boulderlab` is `172.18.10.0/24` with nodes at `172.18.10.1-7`.
Notably the deploy workflow's own `CLUSTER_NODE_CIDR` environment variable holds
the **correct** `10.224.0.0/12`, visible in the v0.2.4 run. The Site and the
workflow are two independently maintained sources of truth and they disagree.
`managedFields` shows `kubectl-unbounded` applied `f:nodeCidrs` at
`2026-08-17T19:16:26Z`. No CI ran that day until 23:58, so this was a manual
`site init`/`install` with `--cluster-node-cidr 10.200.0.0/16`. The value appears
nowhere in this repository, so it came from another cluster's config or a stale
runbook. There is no audit trail: the cluster has **no diagnostic settings at
all**, so `kube-audit` was never enabled.
The product gaps that let this happen silently are filed separately as #649.
**Fix:** re-run the CLI with `--cluster-node-cidr 10.224.0.0/12`. It owns the
field via server-side apply, so patching the object directly would be undone by
the next apply.
## 2. `metalman-controller-boulderlab` cannot schedule
This is what actually fails the release gate:
```
deployment.apps/metalman-controller-boulderlab 0/1 ghcr.io/azure/metalman:v0.2.4
pod/...-56c85d7666-vwpbg 0/1 Pending 5m13s
pod/...-7dbc7765f-lv4tb 1/1 Terminating 4d23h spark-3d37
NotTriggerScaleUp: pod didn't trigger scale-up: 1 node(s) didn't match Pod's node affinity/selector
```
The new pod cannot be scheduled because no node matches its affinity; metalman is
per-Site and pinned via `SiteNodeAffinity`. Its predecessor had been stuck
`Terminating` on `spark-3d37` for nearly five days, which is the signature of a
node the control plane has lost contact with. All seven boulderlab nodes are
currently `Ready=Unknown`, and seven of eight AKS nodes are `Ready=False`.
**This is not something #619 addressed.** That work tolerates a *DaemonSet*
shortfall caused *only* by NotReady nodes (`RELEASING.md`, Degraded clusters). A
Deployment whose pod is Pending because nothing matches its affinity is neither,
so the gate failed correctly.
**It will recur on every release** until boulderlab is fixed, because
`metalman-targets.sh` discovers per-Site metalman deployments from Site spec and
adds them to the gate.
Options: restore a boulderlab node matching the affinity; or set
`spec.components.metalman.enabled: false` on the `boulderlab` Site, which removes
it from the gate cleanly since targets are discovered rather than assumed; or
force-publish each release with a recorded reason.
## Worth doing regardless
Enable `kube-audit-admin` on this cluster. It excludes read-only calls, so volume
and cost are a fraction of full `kube-audit`, and it would have answered "who
changed this" in one query. Right now there is no record of any mutation anyone
has made to this cluster.
Contributor guide
Research direction
Start with internal/net/controller/site_controller.go:2534 and the CLI path for --cluster-node-cidr, then inspect metalman-targets.sh and RELEASING.md for release-gate behavior. Verify the Site CIDR, boulderlab node readiness and affinity, and the deployment target discovery. Done means unbounded-stable assignment is corrected and boulderlab no longer blocks releases, with the chosen operational fix recorded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, bash, go, kubernetes
- Domain
- cloud, devops, infrastructure, release
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100