migrate deployment to FluxCD GitOps with Sealed Secrets, retiring SOPS and the Makefile apply targets
- Dominant language
- Go
- Stars
- 9
- Forks
- 10
- Avg merge
- 16h 55m
- Merged PRs (30d)
- 6
Description
Today deployment is imperative: SOPS/age-encrypted secrets in `deploy/secrets/` decrypted and applied from a workstation via `make sops-apply-*` / `make secrets`, and manifests applied via `make manifests-apply`. This has produced real drift: the live `maintainerd-server-env` secret holds 7 keys while its template renders only 3, so regenerating from the template would silently drop 4 keys.
**Target state:** the cluster runs a FluxCD `Kustomization` reconciling `deploy/manifests/`; secrets are committed as SealedSecret CRs (encrypted one-way against the in-cluster controller's key), so they are safe in git and need no decryption tooling in CI, on laptops, or in Flux. A deploy is a git commit. The Makefile keeps build/test/image targets; the apply/deploy/secrets targets retire.
**Why Sealed Secrets over SOPS here:** no encryption key to distribute to contributors (sealing needs only the public cert; nobody outside the cluster can read values back), and the sealed CRs flow through the same reconcile path as every other manifest. The trade-off is that encryption is one-way — plaintext source of truth moves to the team's existing secure credential store, and the controller's sealing key must be backed up there immediately after install, or the sealed files in git are unrecoverable in a cluster-loss scenario.
**Plan (phased, each phase shippable):**
*Phase 1 — Sealed Secrets*
1. Install the sealed-secrets controller; back up the sealing key to the team credential store **before anything else**.
2. Seal each of the five secrets (`maintainerd-server-env`, `-db-env`, `-web-env`, `-web-bff-env`, `-bootstrap-env`) **from the live cluster secret**, not from the SOPS files, so the captured state is reality (this permanently fixes the 3-vs-7-key template drift). Store each plaintext env in the credential store as the new canonical source.
3. Commit SealedSecrets under `deploy/manifests/`; apply; verify every workload restarts with identical env.
4. Fix `web-image-build`: it currently decrypts a SOPS file at build time to read `NEXT_PUBLIC_BFF_BASE_URL`. That value is baked into the client bundle and was never secret — move it to a plain Makefile/config variable.
5. Delete `deploy/secrets/`, `deploy/templates/`, the `sops-*` Makefile targets, and `scripts/check-sops-secrets.sh`; update `OPS.MD`.
6. Rotate credentials that were SOPS-encrypted (git history retains the age-encrypted blobs forever; anyone holding the age private key can decrypt historical values). The new GitHub token from #147 is already part of this; rotate the remaining tokens and the DB password at the same time.
*Phase 2 — FluxCD*
7. Bootstrap Flux (source-controller + kustomize-controller minimum) with a read deploy key scoped to this repo; add a `GitRepository` + `Kustomization` for `deploy/manifests/`. No decryption stanza needed — secrets are already SealedSecrets from Phase 1.
8. Retire `manifests-apply`, `manifests-delete`, `secrets`, `bootstrap-run`, and the `*-image-deploy` / `mntrd-image-set` rollout targets; document the new flow (commit → Flux reconciles) in `OPS.MD`. Keep `ci-local`, tests, and image build/push targets.
9. Optional later step (separate issue when we get there): Flux image-automation controllers to bump image tags from the registry automatically, replacing manual tag edits; evaluate hardened base images as part of that work.
**Acceptance:** a secret value change and a manifest change each reach the cluster via git commit alone, with no workstation `kubectl apply` or decryption; deleting a reconciled resource by hand gets restored by Flux; the sealing-key backup has been test-restored once.
**Risks:** sealing from live must happen before any template regeneration (drift landmine); one-way encryption means lost sealing key = unrecoverable git secrets (mitigated by step 1 backup + credential-store plaintext); Flux bootstrap gives the cluster pull access to the repo — consider a dedicated deploy repo or path-restricted deploy key if that's a concern.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.