Azure / Azure/unbounded

release-upgrade.yaml hardcodes old namespaces; make it namespace-aware before the first unbounded-system release

Open
#374 0 comments 0 reactions 0 assignees View on GitHub
bug github_actions
Dominant language
Go
Stars
28
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
55

Description

## Summary

`.github/workflows/release-upgrade.yaml` (the unbounded-stable deploy + soak +
publish pipeline) still hardcodes the pre-unification namespaces
(`unbounded-net`, `unbounded-kube`). The namespace unification work (#372)
moved all first-party components onto a single configurable namespace
defaulting to `unbounded-system`, but deferred this workflow because it deploys
**released** artifacts and older releases still install into the old
namespaces.

This must be fixed **before the first `unbounded-system` release is cut**,
otherwise the deploy/soak/publish gate will break for the cutover release.

## Why it is broken

`release-upgrade.yaml` is triggered by `workflow_run` / `workflow_dispatch`, so
the **workflow definition always runs from the default branch**, while the
**artifacts it deploys are tag-pinned**:

- Applied manifests (`MODE=upgrade`) and the embedded plugin (`MODE=init`) come
from the release tag. Old tags -> `unbounded-kube` / `unbounded-net`; the
cutover tag onward -> `unbounded-system`.
- The hardcoded `kubectl -n ` steps come from `main`.

So whatever single namespace `main` hardcodes is wrong for one era. The branch
is already internally inconsistent today:

- deploy waits + summaries + the #235 config merge + orca secret preflight
still target `unbounded-net` / `unbounded-kube`
(lines ~416, 435-437, 454-456, 545-546, 573-574), but
- `hack/release/smoke/core-namespaces-ready.sh:35` was already flipped to
`unbounded-system`.

Neither an old release nor a new release passes the pipeline end-to-end as-is.

## Specific hardcoded references to fix

`.github/workflows/release-upgrade.yaml`:

- L53 - comment: `Secret unbounded-kube/orca-credentials`
- L416 - #235 merge: `kubectl -n unbounded-kube get cm machina-config ...`
- L435 - wait: `kubectl -n unbounded-net rollout status deploy/unbounded-net-controller`
- L436 - wait: `kubectl -n unbounded-net rollout status ds/unbounded-net-node`
- L437 - wait: `kubectl -n unbounded-kube rollout status deploy/machina-controller`
- L454-456 - deploy summary (best-effort image reporting)
- L545-546 - `kubectl -n unbounded-kube get secret orca-credentials` preflight
- L573-574 - orca summary (best-effort)

`hack/release/smoke/core-namespaces-ready.sh`:

- L35 - `NAMESPACES=(unbounded-system)` should come from the workflow contract
so it is correct for both eras.

## Proposed approach: resolve the namespace at runtime

The workflow cannot know at author-time which namespace a given tag uses, so the
correct fix is dynamic resolution rather than a hardcoded literal. Stable,
version-independent selector labels exist for discovery:

- net controller: `app.kubernetes.io/name=unbounded-net-controller`
- net node: `app.kubernetes.io/name=unbounded-net-node`
- machina: `app=machina-controller`
- orca co-locates with machina in both eras (`unbounded-kube` -> `unbounded-system`);
`hack/orca/deploy-stable.sh` is tag-checked-out and already self-consistent.

Plan:

1. **`deploy` job**: add a "Resolve component namespaces" step that discovers
`NET_NS` / `MACHINA_NS` by the labels above (from the live cluster in
upgrade mode; post-apply for init), exports them to `$GITHUB_ENV`, and
publishes them as **job outputs** (`net_ns`, `machina_ns`) next to the
existing `mode`. Optionally fail if a label resolves to >1 distinct
namespace (the "applied across the boundary without migrating" signal).
Use `$NET_NS` / `$MACHINA_NS` in the #235 merge, the rollout waits, and the
summary.
2. **`deploy-orca` job**: consume `needs.deploy.outputs.machina_ns` as
`ORCA_NS` for the secret preflight and summary (or drop the redundant
preflight, since `deploy-stable.sh` already validates the secret in its own
namespace).
3. **smoke**: make `core-namespaces-ready.sh` read `CORE_NAMESPACES`
(default `unbounded-system`) from the contract, and have the `smoke-tests`
job pass it from the dedup of `needs.deploy.outputs.{net_ns,machina_ns}`.

A simpler alternative is a "try both namespaces" loop in each step
(`for ns in unbounded-system unbounded-net; do kubectl -n "$ns" ... && break; done`),
which is also old/new compatible but more repetitive and can mask wrong-namespace
errors. Label discovery + job outputs is preferred.

## Out of scope / hard limitation: the one-time cluster cutover is manual

This makes the **automation** survive the migration window in both directions,
but the **one-time data/namespace cutover stays manual**. Kubernetes namespaces
cannot be renamed in place; a plain `kubectl apply` of a cutover release onto an
old-namespace cluster creates a parallel install (two controllers, split
leader-election leases), and auto-deleting the old namespaces would destroy the
orca/garage PVC.

The unbounded-stable cluster must therefore be migrated once, by hand, around
the first `unbounded-system` release. Recommended ordering (capture this in the
release notes / runbook for that release):

1. **Back up stateful data first.** Snapshot orca/garage object storage and any
PersistentVolumes - they are the only data that cannot be regenerated. The
inventory PostgreSQL lives in the separate `inventory` namespace and is
unaffected.
2. **Recreate operator-owned secrets in `unbounded-system`** (names unchanged):
machina SSH key Secrets (`ssh-`), machine-ops credential Secrets
referenced by `MachineOperationCredential`, and `orca-credentials`.
Example: `kubectl get secret ssh- -n unbounded-kube -o yaml | sed 's/namespace: unbounded-kube/namespace: unbounded-system/' | kubectl apply -f -`
3. **Deploy the cutover release**, which installs into `unbounded-system`.
4. **Validate** rollout/health in `unbounded-system`.
5. **Decommission the old namespaces** once healthy:
`kubectl delete namespace unbounded-kube unbounded-net gantry-system`
(this also releases the old leader-election leases, which follow
`POD_NAMESPACE`).

The dynamic workflow then tracks whatever is installed before and after this
step with no further changes.

## Acceptance criteria

- [ ] No hardcoded `unbounded-net` / `unbounded-kube` namespace references remain
in `release-upgrade.yaml`; namespaces are resolved at runtime.
- [ ] `core-namespaces-ready.sh` derives its namespace set from the workflow
contract.
- [ ] A `workflow_dispatch` deploy of a current (old-namespace) release still
passes deploy + smoke.
- [ ] A deploy of the first `unbounded-system` release (after the manual
cluster cutover) passes deploy + smoke.
- [ ] The manual unbounded-stable cutover ordering (above) is captured in the
release notes for the first `unbounded-system` release.

## Context

Follow-up to #372 (namespace unification). See that PR's description
("Follow-up (not in this PR)") for background. The cutover steps above are the
canonical record; there is no separate design doc.

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/release-upgrade.yaml and hack/release/smoke/core-namespaces-ready.sh, then review the namespace contract from #372 and the existing label selectors. Run a workflow_dispatch for an old-namespace release and inspect the deploy, smoke, and deploy-orca jobs. Done means both old and unbounded-system releases resolve namespaces at runtime, pass deploy and smoke, and the cutover ordering is captured in the first-release notes.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, kubernetes, shell
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.