a2n-seattle / a2n-seattle/rms-app

Provision alpha(scratch)/beta(mainline)/gamma(pilot cities)/production promotion pipeline with PR integration testing and branch protection

Abierto
#350 3 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
TypeScript
Estrellas
1
Forks
1
Merge medio
27 min
PR fusionados (30 d)
4

Descripción

**Scope simplified (2026-08-08):** the elaborate multi-phase design further down in this body (scratch-alpha/force-push, two-tier local integration testing, city registry + fan-out, 7-day soak timer) has been cut down at the user's request. Final direction: `alpha` keeps its current dual role as both the real PR-merge target and the hourly canary target, unchanged -- it just gets real branch protection (currently none exists). `beta` becomes Seattle's real deploy branch (the first pilot city). `master` becomes production for whichever other cities come later (single branch, no per-city fan-out -- that's tracked separately under the multi-tenant epic). All three branches are protected so only an automated GitHub Actions identity can write to them (PR-merge-only for alpha, `promote.yml`'s fast-forward push only for beta/master) -- a "semi-CD" model: promotion stays a manual, human-triggered decision, but the actual branch write is never a bypassable direct push. See the current plan draft in the comment below (``) for the concrete design. The original body below is kept for historical context on the problem this issue is solving, not as the current plan.

Story Points: [8]

# Why?

**As a** repo maintainer
**I want** a real multi-environment promotion pipeline (PR integration testing → alpha → beta canary → gamma pilot cities → production) with branch protection on every stage
**So that** PRs get real integration-test coverage before merge, releases soak through canary and pilot-city stages before hitting all cities, and merges are actually gated by required checks instead of relying on convention (or `gh pr merge --admin`) alone.

This supersedes and absorbs the original, narrower ask below (kept for context): stop using `gh pr merge --admin` and set up real branch protection on `alpha`. Checked at the time: `alpha` had **no branch protection configured at all** (`gh api repos/a2n-seattle/rms-app/branches/alpha/protection` → 404), so `--admin` wasn't actively bypassing anything — but that's exactly the gap this larger plan closes properly, across every stage, not just alpha.

# What?

Today only `alpha` is a real, provisioned AWS environment (one Cognito pool, one Lambda set, 7 DynamoDB tables suffixed `-alpha`, one Amplify Hosting branch), and it plays three overlapping roles at once: the only place `test:integ` can run, the only thing `backend-canary.yml` smoke-tests hourly, and the branch every PR merges into. There is no automated integration-test gate before merge, no environment between "PR merged" and "this is now live in the one real environment," and no branch protection anywhere.

Target end state: PRs get real integration-test coverage against live AWS infra before merge (using `alpha` as a disposable per-PR scratch target, reset to whatever PR is currently under test, not a persistent branch anyone merges into); `beta` becomes the real mainline branch PRs merge into, continuously canary-tested; a manual, canary-gated promotion from `beta` to `gamma` (pilot cities); a manual, 7-day-soak-gated promotion from `gamma` to `master`/production (remaining cities); a registry of what cities exist and what stage each is at; and branch protection across every branch in the chain.

**Key constraint**: the 7 DynamoDB tables are **not CDK-managed** (`Table.fromTableArn` in `storage/tables.ts`, since `ampx` can't import pre-existing tables) — every new environment's tables must be created out-of-band before its first deploy, with no automatic teardown. This is why `alpha` is reused as a shared scratch target (reset per PR) rather than fully ephemeral per-PR infrastructure — provisioning/tearing down real DynamoDB tables per PR was judged disproportionate for this repo's PR volume.

## Target architecture

```
Feature branch ──PR opened against beta──▶ CI (unit tests, existing)

└─▶ NEW: local/mocked integration-test tier
(DynamoDB Local or similar, no real AWS) must
pass first, THEN force-push PR HEAD onto `alpha`
(git push --force origin HEAD:alpha)
→ triggers alpha deploy (ampx pipeline-deploy)
→ runs a BASIC e2e smoke suite against alpha
(small subset of real Cognito-sign-in +
Lambda-invoke checks, not the full test:integ
battery)
→ status check reported back on the PR
(serialized: concurrency group so only one
PR's content occupies alpha at a time)

PR merged into beta (only after check is green)


beta = mainline branch, own Cognito pool,
own 7 `-beta` tables, own Amplify branch,
continuously canary-tested hourly (full test:integ
suite runs here on schedule, not per-push)

promote.yml: beta → gamma (manual workflow_dispatch,
BLOCKED unless beta's latest canary run succeeded —
checked automatically, not just trusted)


`gamma` = canonical staging branch (fast-forwarded
from beta; not itself deployed anywhere)

fan-out: for every city in the registry currently
at "gamma" stage, fast-forward that city's own
branch (e.g. `seattle`) to match `gamma`'s
new HEAD → each triggers that city's own Amplify
deploy, real user traffic, still closely watched

promote.yml: gamma → master (manual workflow_dispatch,
BLOCKED until 7 days have elapsed since that city's
content landed in gamma — enforced via a GitHub
Environment wait timer, not a home-rolled date check)


`master` = canonical production branch (fast-forwarded
from gamma; not itself deployed anywhere)

fan-out: for every city in the registry currently
at "production" stage, fast-forward that city's
own branch (e.g. `vancouver`) to match
`master`'s new HEAD → each triggers that city's
own Amplify deploy
```

`alpha` stops being a branch anyone merges into or reads history from — it's a scratch AWS deploy target, force-pushed on every PR push. `beta` takes over alpha's old role as "the" mainline branch. This is a real change from CLAUDE.md's current standing rule ("`alpha` is this repo's default branch and the merge target for all PRs") — updating that doc, the repo's actual GitHub default-branch setting, and the issue-pipeline's "branch from `origin/alpha`" instruction (should become `origin/beta`) are part of Phase 1.

## City registry

Add a single source of truth for what cities exist and what stage each is at — e.g. `amplify/config/cities.json`. A city's branch is just its own name (no `gamma-`/`production-` prefix) — the registry's `stage` field, not the branch name, tracks whether it's currently a pilot or fully in production, since the same branch (e.g. `seattle`) gets fast-forwarded regardless of which stage it's at.

**Seattle is the real, decided first pilot city** — Phase 3 actually provisions and releases to it as part of this work, not just as a naming example. `vancouver` in the snippet below is purely illustrative of the registry's `"production"`-stage shape (no production city has been decided yet):

```json
{
"cities": [
{ "name": "seattle", "stage": "gamma", "amplifyBranch": "seattle", "tableSuffix": "seattle", "promotedToGammaAt": "2026-08-01T00:00:00Z" },
{ "name": "vancouver", "stage": "production", "amplifyBranch": "vancouver", "tableSuffix": "vancouver", "promotedToProductionAt": "2026-05-01T00:00:00Z" }
]
}
```

`promote.yml`'s fan-out step reads this on every `gamma`/`master` promotion: after fast-forwarding the canonical `gamma` (or `master`) branch, it iterates every entry whose `stage` matches ("gamma" or "production") and fast-forwards that entry's `amplifyBranch` to the canonical branch's new HEAD — so adding a new pilot city later is a one-line registry addition plus provisioning its tables/Cognito pool, not a workflow-file edit. When a city graduates from pilot to full production, only its `stage` flips (fed from `master` instead of `gamma`) — its branch name never changes.

## Branch protection (reset from scratch, all branches)

| Branch | Who can push | Force push | Required checks | Notes |
|---|---|---|---|---|
| `alpha` | CI workflow identity only (bot/PAT used for the force-push step) | Allowed, but only for that same identity | none (not a review target) | Humans blocked from pushing/force-pushing directly; this is the one branch where the automation's force-push is expected and everything else's isn't. |
| `beta` | Anyone, via PR only | Blocked | Alpha smoke-test status check + required reviews | This is the new default/mainline branch — becomes the repo's GitHub default branch and the issue-pipeline's `origin/beta` base. |
| `gamma` (canonical) | promote.yml's identity only | Blocked | GitHub Environment `gamma` gate: automated "beta canary succeeded" check | Not itself deployed anywhere — exists so there's one clear fast-forward point from `beta`, and one clear diff/history of what's been approved to pilot. |
| `master` (canonical) | promote.yml's identity only | Blocked | GitHub Environment `master` gate: 7-day wait timer (GitHub Environments support wait timers up to 30 days — a native fit, not custom date-diffing logic) | Manually triggered, but the job can't actually run its fast-forward until the timer elapses. |
| `` (e.g. `seattle`, `vancouver`) | promote.yml's identity only | Blocked | none beyond whichever canonical branch's gate last fed it | One branch per city, no stage prefix — fast-forwarded by the fan-out step from `gamma` while the registry marks it `"stage": "gamma"`, or from `master` once it's `"stage": "production"`. This is what Amplify actually deploys per city. |

# How?

## Phase 1 — Reprovision alpha as scratch/PR-test target, beta as mainline

- Flip the repo's default branch and PR target from `alpha` to `beta`. Update CLAUDE.md's "Branching and environment promotion" standing rule and the issue-pipeline's "branch from `origin/alpha`" instruction to say `beta`.
- New workflow triggered on `pull_request` (opened/synchronize) against `beta`: runs the new local/mocked integration-test tier first (see below); only if that passes does it force-push the PR's HEAD onto the `alpha` ref, deploy (`ampx pipeline-deploy --branch alpha`), and run the basic e2e smoke suite directly in the same job (so the check attaches to the PR naturally, rather than relying on `backend-cd.yml`'s separate `push`-triggered run). Add `concurrency: group: alpha-integ-test, cancel-in-progress: false` so only one PR occupies alpha at a time; others queue rather than racing or being cancelled mid-deploy.
- Set up branch protection per the table above for `alpha` and `beta`.
- Move `backend-canary.yml`'s hourly smoke test from `alpha` to `beta` (beta is now the thing that needs continuous validation; alpha's content is transient and not meaningful to canary).
- **Two-tier test gate, to make each alpha release count**: split what runs where instead of firing the full real-AWS `test:integ` suite on every push.
1. **Before any push reaches alpha** — a new local/mocked integration-test tier (e.g. DynamoDB Local or a similar in-process substitute for the DB-layer, plus the existing unit tests) runs in PR CI with no real AWS deploy at all. This needs new build-out: `test:integ` today can't run fully offline (real Cognito + Identity Pool + live Lambda invoke per `Amplify.test.ts`), so this tier realistically covers `db/*.ts`/`api/*.ts` business logic against a local DynamoDB substitute, not the Cognito-authenticated invoke path — scope that precisely during implementation. This tier must pass before the PR branch is force-pushed to `alpha` at all.
2. **Only after that tier is green** does the PR workflow force-push to `alpha`, deploy, and run a **basic end-to-end smoke suite** against the real environment — a small, fast subset of real Cognito-sign-in + Lambda-invoke checks (not the full `test:integ` battery), just enough to catch "the real deploy is broken" issues the local tier can't see. Keep the full `test:integ` suite for the hourly canary on `beta`, where a fuller sweep on a schedule is cheaper than on every push.
- **Update CLAUDE.md with a "minimize alpha pushes" rule**: every push to a PR branch that clears tier 1 still costs a real alpha deploy + smoke-suite run (real AWS calls, queued behind the `alpha-integ-test` concurrency group behind any other PR currently testing), so pushing early/often as a substitute for local iteration is expensive, not free, under this design. Document that contributors should run `npm run build && npm run test:unit` (and the new local/mocked integration tier) locally until confident, and only push to the PR branch — triggering a real alpha run — once ready, not use alpha as the primary feedback loop during development.
- Stop passing `--admin` to `gh pr merge` (the original ask this issue supersedes) — once `beta` has real required checks/reviews, `--admin` would start actively bypassing them.
- **Test coverage**: workflow/config change, not `ts-code/` logic, except for the new local/mocked integration-test tier itself, which is new test infrastructure and needs its own coverage (at least one representative test proving it actually catches a DB-layer regression) — verified by a real PR going through the new two-tier gate (and by intentionally breaking each tier on a throwaway commit to confirm each blocks appropriately, then reverting).

## Phase 2 — Provision `beta` as the canary-tested mainline environment

- Parameterize `storage/tables.ts`'s `envSuffix` (hardcoded `"alpha"` today) by branch name so a `beta` deploy resolves to `main-beta`, `items-beta`, etc.
- Manually create the 7 `-beta` DynamoDB tables (schema from `ts-code/src/db/Schemas.ts`) before first deploy; document the exact `aws dynamodb create-table` commands so the same steps repeat for every gamma/production city.
- Add `beta` as an Amplify Hosting branch (own Cognito pool + Lambda set, its own CloudFormation stack).
- `promote.yml` already needs no change here (alpha→beta source mapping already exists); the new piece is the automated canary-success check gating `beta → gamma`, added in Phase 3.

## Phase 3 — `gamma`: pilot cities, canary-gated promotion from beta

- Add Seattle to `amplify/config/cities.json` as the first `stage: "gamma"` entry — **this is a real release target for this issue, not just an example**: Seattle's `seattle` branch/tables/Cognito pool actually get provisioned and promoted to as part of this work. Confirm with the user before provisioning any additional pilot cities beyond Seattle.
- Provision Seattle's environment the same way as beta: own tables/Cognito/Lambda, tables suffixed per the registry.
- Extend `promote.yml`: when target is `gamma`, before fast-forwarding the canonical `gamma` branch, query the beta canary workflow's most recent run (`gh run list --workflow= --branch beta -L1 --json conclusion`) and fail the promotion outright if that run didn't succeed — "canary is ok" becomes a checked precondition, not a trusted human judgment call.
- After the canonical `gamma` branch is fast-forwarded, add the **fan-out step**: read `amplify/config/cities.json`, and for every entry with `stage: "gamma"`, fast-forward that entry's `amplifyBranch` (e.g. `seattle`) to `gamma`'s new HEAD. This is what actually triggers each pilot city's Amplify deploy — the canonical `gamma` branch itself is never an Amplify Hosting branch.
- Define a `gamma` GitHub Environment (one, not per-city) and require the promote job's canonical fast-forward step to target it, so the canary-check gate is enforced by GitHub's deployment protection rules rather than only by the workflow's own logic.
- Suggested cadence: promote `beta → gamma` **weekly**, batching whatever merged that week.
- Soft, non-enforced norm: avoid merging discretionary work to `beta` during the first week of each month, to let whatever's about to go to production settle before the monthly `gamma → master` promotion. Urgent bug fixes still ship on their own schedule regardless.

## Phase 4 — `master`/production: remaining cities, 7-day-soak-gated promotion from gamma

- Same per-city table/Cognito/Lambda provisioning pattern, one environment per remaining city, added to the registry with `stage: "production"` as each is promoted.
- Define a `master` GitHub Environment (one, not per-city) with a 7-day wait timer deployment protection rule. `promote.yml`'s `gamma → master` path targets this environment for the canonical fast-forward; triggering it starts the countdown, and the job's fast-forward step only executes once the timer clears — this directly implements "changes must sit in gamma for a week before release," using a GitHub-native mechanism instead of custom date tracking.
- After `master` is fast-forwarded, run the same **fan-out step** pattern as Phase 3: for every registry entry with `stage: "production"`, fast-forward that entry's `amplifyBranch` (e.g. `vancouver`) to `master`'s new HEAD, triggering that city's Amplify deploy.
- Remove/adapt `promote.yml`'s current hard-fail-on-`master` guard once the first city's gamma→production path has been proven end-to-end.
- Suggested cadence: promote `gamma → master` **monthly**, not weekly — it can't legally beat the 7-day soak floor anyway, and since this promotion's build cost scales with the *full* city count (eventually much larger than the pilot set), monthly keeps that cost to roughly a quarter of what weekly would cost as the city count grows. Batch all eligible cities into one promotion event rather than promoting per-city on separate schedules.

## AWS free-tier feasibility

Lambda, DynamoDB, and Cognito can plausibly stay free or near-free even as environments multiply, since their free tiers are always-free and mostly account-wide (Lambda/DynamoDB) or per-pool (Cognito):

- **Lambda**: 1M requests + 400,000 GB-s/month, always-free, account-wide across every environment combined. Current alpha canary is ~10 invocations/hour (~7,200/month); each new environment's hourly canary adds a similar small amount. Even at 10+ environments all canaried hourly, total stays roughly two orders of magnitude below the 1M ceiling.
- **DynamoDB**: 25GB storage + 25 RCU/25 WCU always-free, account-wide if using provisioned capacity (shared across every environment's tables); if on-demand billing is used instead, cost is expected in the cents/month range even across many environments at this app's request volume.
- **Cognito**: each environment gets its own User Pool, and the Essentials tier's 50,000 MAU free allowance is *per pool* — multiplying environments multiplies free headroom rather than consuming a shared pool. Stays free as long as advanced security features aren't enabled.
- **Amplify Hosting is the real exception**: its free tier (1,000 build-minutes/month, 5GB served) is time-limited to the account's first 12 months, not always-free. Every environment is a new Amplify branch rebuilding on every push/promotion, so build-minute consumption scales roughly with environment count × promotion frequency. Post-12-months this becomes small pay-as-you-go (~$0.01/build-minute, ~$0.023/GB stored, ~$0.15/GB served) — likely low single-digit dollars/month at this scale, but not literally $0. The weekly/monthly promotion cadences above are chosen specifically to keep this line item's growth manageable as city count grows.
- **CloudWatch Logs** (5GB ingest + 5GB storage/month, always-free, account-wide) is the other line item likely to tip into paid territory first, since it's shared across every environment's Lambda logs with no visible retention policy today. Recommend explicit log-group retention (14-30 days) as Phase 1 hygiene before environment count grows.

Bottom line: the core compute/data/auth services can realistically stay near-free through the pilot-cities stage and likely well beyond it; Amplify Hosting build-minutes and CloudWatch log retention are the two components to flag as incurring small real cost rather than promising the whole pipeline is free forever.

# Additional context

Supersedes #350 ("Stop using `gh pr merge --admin`; set up real branch protection on alpha"), which is now fully covered as a subset of Phase 1's branch-protection work above.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

El issue describe un pipeline complejo de promoción con múltiples ramas. Empieza leyendo los workflows existentes de GitHub Actions del repositorio (como .github/workflows/backend-cd.yml) y la configuración de amplify. Comprende el despliegue actual de alpha y la configuración de la tabla de DynamoDB en storage/tables.ts. El objetivo es implementar la protección de ramas, un registro de ciudades en amplify/config/cities.json y nuevos workflows de promoción. «Done» significa un pipeline funcional en el que los PRs activan pruebas de integración, beta es la mainline y las promociones a gamma y master están protegidas.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
aws, docker, github-actions, nodejs
Área
backend, ci-cd, cloud, devops, infrastructure
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
25/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.