aws-samples / aws-samples/sample-autonomous-cloud-coding-agents
feat(cdk): stop integ harness from surfacing a misleading NEW-snapshot failure
- Dominant language
- TypeScript
- Stars
- 143
- Forks
- 46
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 20
Description
### Component
CDK / infrastructure (Tooling / CI — integ harness)
### Describe the feature
Stop the integ smoke harness from carrying an integ-runner **snapshot** that provides no signal and surfaces a misleading `1 failed` line on every run. Either remove the snapshot phase from the workflow's mental model by documenting it as expected, or (preferred) reshape the harness so integ-runner no longer reports a spurious failure — so a reader of a run log isn't led to believe the smoke test failed when it passed.
### Use case
In run [28892156232](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/actions/runs/28892156232/job/85707381514) the log reads:
```
Verifying integration test snapshots...
NEW integ.task-api-smoke 0.002s
Tests: 1 failed, 1 total ← snapshot phase: "NEW" (no baseline)
Running integration tests for failed tests...
SUCCESS integ.task-api-smoke-TaskApiSmoke/DefaultTest 291.541s
Tests: 1 passed, 1 total ← authoritative result
```
The `1 failed` is not a test failure — it is integ-runner classifying the test as `NEW` because there is **no committed snapshot to diff against**. `/cdk/test/integ/*.snapshot/` is gitignored on purpose (each snapshot bundles ~100 MB of Lambda handler assets), and the harness runs `npx integ-runner --force` (`cdk/mise.toml:69`), which deploys-then-verifies unconditionally and ignores any snapshot verdict.
So the snapshot's normal job — an offline template diff that lets CI skip the deploy when the stack is unchanged — is structurally disabled here. It can never match, so it always reports `NEW`/failed, then always falls through to the live run. Net signal today: zero; net effect: a confusing red-looking line in every run. This repeatedly misleads anyone triaging a run (it's easy to conflate the snapshot line with the real job failure, which in that run was the separate `Ensure stack torn down` step — see #566).
Note the path filter (`^(cdk|agent)/` in `integ.yml`) gates *whether the job runs at all*, but is coarser than the snapshot (which would gate on the specific stack's synthesized template). The two are not equivalent — but given the always-`--force` design, the snapshot's finer check is thrown away regardless, which is why it can go.
### Proposed solution
Pick the lightest option that removes the misleading output:
1. **Suppress / accept the NEW-snapshot noise (minimal):** keep the current always-deploy model but add a short comment near the integ step and in `cdk/mise.toml` stating that integ-runner reports the uncommitted snapshot as `NEW`/`1 failed` on every run and the authoritative result is the live deploy-then-verify. Optionally post-process integ-runner output so the job summary shows only the live-run result.
2. **Commit a pruned, template-only snapshot (restores the offline gate):** regenerate the snapshot without bundling Lambda assets so it is small enough to commit, giving back the "skip deploy when the stack template is unchanged" optimization the snapshot exists for. More work; only worth it if we want to cut real deploys.
Recommendation: option 1 — the smoke stack is tiny and every applicable PR does a real deploy anyway, so the offline gate buys little and committing assets was already rejected as too heavy.
Out of scope: changing the deploy → assert → destroy flow itself, and the teardown/naming work in #566 / #400.
### Other information
- Related: #566 (removes the broken teardown safety net in the same workflow; the snapshot line and the teardown failure are the two separate red herrings in run 28892156232), #400 (integ harness naming/cleanup), #236 (Phase-0 parent).
- Relevant paths: `.github/workflows/integ.yml`, `cdk/mise.toml` (`[tasks.integ]`), `.gitignore` (`/cdk/test/integ/*.snapshot/`), `cdk/test/integ/integ.task-api-smoke.ts`.
Contributor guide
Research direction
Start with cdk/mise.toml [tasks.integ] and .github/workflows/integ.yml, then inspect .gitignore and cdk/test/integ/integ.task-api-smoke.ts. Run the integ harness with its current npx integ-runner --force behavior and compare the snapshot and live-run output. Done means the run no longer presents the expected NEW snapshot as a misleading test failure, while the live deploy-and-verify result remains authoritative.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, github-actions, typescript
- Domain
- ci-cd, cloud, testing, tooling
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100