aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

feat(ci): anchor cdk diff to last-authorized baseline (git-tagged main-* deploy) + surface drift at the deploy gate

Open
#383 2 comments 0 reactions 0 assignees View on GitHub
ci-cd enhancement infra-cdk security
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

### Primary area

CI/CD + Security (deploy approval pipeline)

### Summary

The pre-approval `cdk diff` that informs the `deploy` environment approver compares the candidate template against **whatever is currently deployed in the account** (`cdk diff --method=template`, `deploy.yml:173-175`), not against the **last authorized (merged-to-`main`) baseline**. Any drift — console edit, hotfix, partial/rolled-back deploy, another pipeline — gets silently absorbed into that live baseline and shows the approver "no change." At the one checkpoint where a human is asked "is this safe to deploy?", they cannot tell "approved last week" from "changed by hand on Tuesday."

**Fix:** diff the candidate synth against a baseline **synthesized from `main`**, supplied as a local template file. This is deterministic, authorized-by-merge, and fully offline (no CloudFormation read role). The baseline already exists as a CI artifact on every merge — no new synth, no new permissions.

### Current mechanism (grounded in `deploy.yml`)

The `diff` job (`.github/workflows/deploy.yml:129-198`) runs before the `deploy` gate:

```yaml
# deploy.yml:173-175
# --method=template: read-only comparison against deployed template;
# no change-set creation, no S3 asset publishing, no deploy role needed.
npx cdk diff --app cdk/cdk.out --all --method=template --no-color ...
```

It uses a read-only IAM role on the `diff` environment (`deploy.yml:158`), writes full + `--security-only` diffs to `$GITHUB_STEP_SUMMARY`, and uploads them as artifacts. Surfacing changes to the approver is correct and necessary — the **baseline** is the flaw, not the plumbing.

### The gap

`cdk diff --method=template` (no `--template`) diffs against the **live deployed template** per stack. That is "current reality," which is not a trustworthy authorized baseline: there's no record of which template state a human last approved, so account drift becomes the implicit baseline and reads as "no change" in the approver's diff. Drift is invisible at exactly the moment it matters.

### Proposed mechanism: diff against a baseline synthesized from `main`

`cdk diff --template ` compares a stack against a **local CloudFormation template file** and **implies `--method=template`** ([CDK v2 `cdk diff` ref](https://docs.aws.amazon.com/cdk/v2/guide/ref-cli-cmd-diff.html)). Feed it `main`'s synthesized template as the baseline and the diff reads **nothing** from the account: no `cloudformation:GetTemplate`/`DescribeStacks`, no lookup/deploy role — it runs with **no AWS credentials**. The baseline is deterministic (what `main` synthesizes is what was reviewed and merged), so drift can't rebaseline it.

#### Baseline source: the `merge_group` build artifact (no synth, no cache)

Every merge to `main` runs `build.yml` on the **`merge_group`** event (`build.yml:7`) — a **required** check, so the artifact is guaranteed to exist for every merged commit. That build uploads, per `compute_type` in the matrix (`build.yml:51-52`, `295-301`):

```yaml
# build.yml:295-301
- name: Upload CDK artifact (${{ matrix.compute_type }})
uses: actions/upload-artifact@...
with:
name: cdk-${{ matrix.compute_type }}-out
path:
cdk/cdk.out/ # synthesized templates ← the baseline
cdk/cdk.context.json # resolved context ← no re-synth, no lookups
```

Example: run [27726107065](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/actions/runs/27726107065) (`merge_group`, success) → artifact `cdk-agentcore-out` (~96 MB, expires 2026-09-15).

So the baseline for the review diff is: **download `cdk--out` from the most recent successful `merge_group` `build` run on `main`, use its `cdk.out/.template.json` as the `--template` baseline.** Properties this gives us:

- **No re-synth.** The artifact already contains synthesized templates. We never check out `main` and `cdk synth` — and because `cdk.context.json` is bundled, even if we did, there are no account lookups.
- **90-day retention is plenty.** Artifact retention default is 90 days; at our merge cadence the latest authorized baseline is always well inside that window. (Run 27726107065's artifact expires 2026-09-15, ~90 days out.)
- **Do not use Actions cache for this.** Cache is 7-day eviction and a 10 GB repo cap — untrustworthy as an authorization baseline (could be evicted, could be poisoned by any branch). Artifacts are the source of record; cache is not.
- **Provenance is the build's commit.** The baseline's authorized commit is the `head_sha` of the `merge_group` run the artifact came from — record it in the approver's summary ("baseline = `cdk-agentcore-out` from run X, commit `abc1234`").

#### Sequence

1. **Resolve baseline:** find the most recent successful `build` run on `main` via the `merge_group` event; download its `cdk--out` artifact → `baseline/cdk.out/`.
2. **Diff candidate vs. baseline, per stack** (template-to-template, offline):
```bash
npx cdk diff "$STACK" --app cdk/cdk.out --template "baseline/cdk.out/$STACK.template.json" --no-color
```
3. **Report:** full diff to artifact + step summary; re-run with `--security-only` for the focused IAM / security-group-broadening view the approver gates on. Lead the summary with security changes.

### Implementation wrinkles (build-ready)

- **`--template` is per-stack.** It takes one template file and does not combine with `--all`. Loop over stacks, pairing each candidate to its `baseline/cdk.out/.template.json`. Handle **added** stacks (no baseline file → all-new) and **removed** stacks (baseline file, no candidate → all-removed) explicitly.
- **Matrix.** Pair baseline + candidate per `compute_type` (the job is already a matrix; the artifact name is per-`compute_type`).
- **First-run / missing baseline.** If no prior `merge_group` artifact exists (cold start) or it expired, fall back to the deployed-template diff (`--method=template` against live) with a loud "no authorized baseline available — comparing against live state" note, so the gate degrades gracefully instead of failing.
- **Asset-hash / cosmetic noise.** `--method=template` renders replacement-triggering property changes as replacements even when cosmetic, and asset-hash bumps show as diffs. `--security-only` sidesteps most of this for the gate-relevant view; the full diff stays informational.
- **No read role on the happy path.** The synth-from-`main` review diff needs no AWS creds. Keep the read-only role only for the optional live-drift probe below.

### Optional second signal: drift probe (account vs. authorized)

Distinct from the review diff and **not** the baseline for change review:

- **`baseline(main) → candidate(PR)`** — the review diff (above). Offline, deterministic, no role. The approver's primary artifact: "what is this change introducing vs. the authorized baseline?"
- **`baseline(main) → live deployed`** — an optional, clearly-labeled drift probe (needs the read-only role): "has the account drifted from the last authorized deploy?" If non-empty, surface it loudly; decide later whether it flags or blocks the gate.

### Acceptance criteria

- [ ] Pre-approval diff compares candidate synth against a baseline **synthesized from `main`**, supplied via `cdk diff --template`, not against live account state.
- [ ] Baseline is sourced from the `cdk--out` artifact of the most recent successful `merge_group` `build` run on `main` (contains `cdk.out/` + `cdk.context.json`); **no re-synth**, **no Actions cache**.
- [ ] Per-stack loop handles added/removed stacks; per-`compute_type` matrix pairing.
- [ ] Review diff runs with **no AWS credentials**; read-only role retained only for the optional live-drift probe.
- [ ] Approver summary leads with `--security-only` changes and records baseline provenance (artifact run id + commit `head_sha`).
- [ ] Graceful fallback to live-template diff (with a loud note) when no authorized baseline artifact is available.
- [ ] (Optional) `baseline → live` drift probe surfaced as a separate, labeled signal.

### Open questions

- Drift probe: **flag** or **block** the gate on non-empty drift? (Lean: flag first, block once drift is rare.)
- Baseline resolution: query the latest `merge_group` run via the Actions API at diff time, or have `deploy.yml` consume a pinned run id? (API lookup is self-maintaining; pinned id is auditable.)
- Should the baseline be the latest `main` merge, or the last *deployed* `main` commit specifically (if those can diverge under #377's split)?

### Backlog relationships

- **#120 (RFC: Least-privilege CDK bootstrap policies as code)** — the current `diff` job's read-only role lives in this area. This proposal **removes** the role from the review-diff happy path (offline synth-from-`main`); the read-only role is needed only for the optional live-drift probe. Coordinate that reduced scope here.
- **#377 (RFC: Separate infrastructure deploy from application/runtime updates)** — the deploy split changes cadence and what each lifecycle deploys; the "baseline = last `main`" definition must account for infra vs. runtime vs. config layers (see the third open question). Design against the post-split topology.

Contributor guide

Open the contributing guide

Research direction

Start with the diff job in .github/workflows/deploy.yml:129-198 and the CDK artifact upload in build.yml:295-301; trace how the matrix and merge_group artifacts are currently handled. Define the baseline-resolution and per-stack comparison flow, including added or removed stacks and the missing-artifact fallback. Done means the acceptance criteria are met, including offline review diffs, security-only reporting, and baseline run and commit provenance.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions
Domain
ci-cd, cloud, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.