aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

feat(cdk): integ harness — ephemeral stack name + offline cdk-diff gate + non-blocking stranded-stack cleanup

Open
#400 0 comments 0 reactions 0 assignees View on GitHub
ci-cd cost-finops enhancement infra-cdk validation-loop
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

### Component

CDK / infrastructure, Tooling / CI

### Summary

Make the integ smoke harness (`.github/workflows/integ.yml` + `cdk/test/integ/integ.task-api-smoke.ts`) **self-healing under unreliable teardown** by giving each run a unique, ephemeral CloudFormation stack name, and give the `integ` environment approver the same **pre-run change visibility** the `deploy` gate already has — via an **offline `cdk diff`** against the last-authorized `main` baseline (no AWS credentials, mirroring #383's mechanism).

Three coupled changes:

1. **Dynamic/unique stack name** — replace the single hardcoded `backgroundagent-integ` with `int-{short-sha}-{run-number}` (or equivalent run-unique id), read at synth time.
2. **Offline `cdk diff` for the integ approver** — surface what the run will create/change in the `integ` environment summary *before* the privileged role runs, diffed against `main`'s synthesized template (the `cdk--out` build artifact), so an approver can see the security posture / resource delta without an AWS read role.
3. **Non-blocking stranded-stack cleanup** — best-effort teardown that does **not** fail the run (or head-of-line block the next run) on the AgentCore `DELETE_FAILED`; leave the stranded stack for the scheduled sweep (#72).

### Motivation / problem

The current harness hardcodes one stack name and serializes all runs:

```yaml
# integ.yml
concurrency:
group: cdk-integ
cancel-in-progress: false # only one run at a time against the shared account
```
```ts
// cdk/test/integ/integ.task-api-smoke.ts:93
const stack = new TaskApiSmokeStack(app, 'backgroundagent-integ');
```
```yaml
# integ.yml "Ensure stack torn down"
aws cloudformation delete-stack --stack-name backgroundagent-integ || true
aws cloudformation wait stack-delete-complete --stack-name backgroundagent-integ # NO `|| true`
```

**Three failure modes:**

- **Teardown is out of our control and fails ~100% of the time.** Per #111, `BedrockAgentCore::Runtime` holds ENI leases in the VPC for up to ~8h after the runtime is deleted; there is no public force-release API. `stack-delete-complete` against a stack whose dependent VPC/SG can't delete ends in `DELETE_FAILED`. (The current smoke stack omits AgentCore to dodge this, but Phase 1 / #317 scenarios that exercise the runtime will hit it directly.)
- **Single fixed name + serialized concurrency = head-of-line block.** A stranded `DELETE_FAILED` stack under the one fixed name blocks every future integ run: the next run can't `CREATE` a stack name that already exists in a failed state, and `cancel-in-progress: false` queues runs behind it. Teardown is "tens of hours and out of our control," so this stalls the required `integ-smoke` check indefinitely.
- **The `integ` approver is flying blind.** Approving the `integ` environment authorizes fork-authored test code to run with the privileged role, but unlike `deploy.yml`'s gate there is **no `cdk diff`** surfaced first — the approver can't see what resources / security posture the run introduces.

### Proposed solution

#### 1. Ephemeral stack name (read at synth time)

- Parameterize `TaskApiSmokeStack`'s id from an env var (e.g. `INTEG_STACK_NAME`), defaulting to a stable local name (`backgroundagent-integ-local`) so the local `mise //cdk:integ` path is unchanged.
- In `integ.yml`, compute `int-{short-sha}-{run-number}` and pass it in. Reuse `build.yml`'s `sanitize()` rules (lowercase; `/_.`→`-`; must start with a letter; CFN length cap) so the name is always CFN-valid.
- Relax the concurrency comment/mitigation: unique names mean overlapping runs no longer collide on one name. (Keep a concurrency group if we still want to bound shared-account cost/parallelism, but it's no longer a *correctness* requirement.)
- The `int-` prefix marks it ephemeral for #72's sweeper (it already treats non-protected stacks as ephemeral).

#### 2. Offline `cdk diff` for the integ approver (mirror #383)

- Before the gated `integ` job, add a diff step (or a preceding job) that downloads the `cdk--out` artifact from the most recent successful `merge_group` `build` run on `main` and diffs the candidate integ template against it:
```bash
npx cdk diff "$STACK" --app cdk/cdk.out --template "baseline/cdk.out/$STACK.template.json" --no-color
```
- This is **fully offline** — no `cloudformation:GetTemplate`, no read role, no AWS creds (the recommended `main--cdk.out` baseline). Surface full + `--security-only` to the step summary so the approver sees the resource/security delta before approving.
- Cold-start / missing-baseline fallback: note loudly and either skip the diff or fall back to live-template diff, matching #383's degradation behavior.
- **Open question:** the integ stack is a *trimmed* app (Task API + tables, no orchestrator/AgentCore) synthesized from `cdk/test/integ/`, so the `main` baseline artifact (`src/main.ts` synth) won't contain a matching `int-*` template. Resolve whether the baseline should be (a) the integ stack synthesized from `main` (requires synthesizing the integ app on `main`, not just the prod app), or (b) diff candidate-integ vs. *previous-integ* run, or (c) a repo-checked `main-integ-cdk.out` reference template. Pick the deterministic, credential-free option.

#### 3. Non-blocking stranded-stack cleanup

- Replace the hard `aws cloudformation wait stack-delete-complete` (no `|| true`) with best-effort teardown that **records but does not fail** on `DELETE_FAILED`/timeout. A stranded ephemeral stack must not fail the integ result or block the next run — that's exactly what #72's scheduled sweep + #111's ENI-aware retain pattern exist to reclaim.
- Optionally adopt #111's `delete-stack --retain-resources` + tag-schema so the VPC/SG are retained and tagged for the sweep instead of leaving a `DELETE_FAILED` stack.

### Relationship to existing issues

- **#317** (Phase 1 core lifecycle, *approved*, assigned to @ayushtr-aws) — its design constraints currently **mandate** the hardcoded `backgroundagent-integ` name and cite single-concurrency as the collision mitigation. This issue **amends that decision**; #317 inherits the harness but its deliverable is the test *scenarios*, not the naming/teardown plumbing. Coordinating so the two don't conflict.
- **#236** (Phase 0 parent) — already anticipates this: *"Prefer a dedicated integ stack name **or ephemeral account** to avoid colliding with developer stacks."* Ephemeral-per-run is inside its stated intent.
- **#72** (scheduled ephemeral cleanup, P1) — the asynchronous reclaim path that makes "don't fail on `DELETE_FAILED`" safe. The `int-` prefix participates in its sweep.
- **#111** (AgentCore ENI cleanup docs) — root cause of the ~8h lease / 100% `DELETE_FAILED`; source of the retain-resources + tag-schema pattern.
- **#383** (offline `cdk diff` against authorized baseline) — the diff mechanism for ask #2 is the same artifact-sourced, credential-free `--template` diff, scoped there to `deploy.yml`. This issue applies it to the `integ` gate; coordinate so both reuse one helper.

### Acceptance criteria

- [ ] Integ stack name is run-unique (`int-{sha}-{run}`) and CFN-valid; local `mise //cdk:integ` path unchanged (stable default name).
- [ ] A stranded `DELETE_FAILED` stack under a prior run's name does **not** block or fail a subsequent integ run.
- [ ] The `integ` environment approver sees a `cdk diff` (full + security-only) in the step summary **before** approving, produced **without AWS credentials**.
- [ ] Teardown is best-effort: `DELETE_FAILED`/timeout is recorded (warning + leaves it for #72) but does not fail the `integ-smoke` status.
- [ ] `integ.yml` comments and #317's design notes updated to reflect the new naming/concurrency model.
- [ ] Docs/ROADMAP synced if any guide references the integ harness behavior (`mise //docs:sync`).

### Out of scope

- Building #72's sweeper Lambda (this issue only ensures it has ephemeral `int-*` stacks to reclaim).
- #383's `deploy.yml` baseline work (separate; share the helper).
- Phase 1 test scenarios themselves (#317).

### Acknowledgements

- [x] I may be able to implement this feature
- [ ] This might be a breaking change

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/integ.yml and cdk/test/integ/integ.task-api-smoke.ts, then compare build.yml's sanitize() rules and #383's artifact-based diff flow. Resolve the stated baseline choice for the trimmed integ app, implement run-unique naming, offline diff visibility, and non-blocking cleanup, then verify the acceptance criteria and run mise //docs:sync if documentation references change.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions, typescript
Domain
ci-cd, cloud, infrastructure, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.