aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

feat(cdk): stop integ harness from surfacing a misleading NEW-snapshot failure

Open
#569 0 comments 0 reactions 0 assignees View on GitHub
ci-cd enhancement infra-cdk tooling
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

### Component

CDK / infrastructure (Tooling / CI — integ harness)

### Describe the feature

Stop the integ smoke harness from carrying an integ-runner **snapshot** that provides no signal and surfaces a misleading `1 failed` line on every run. Either remove the snapshot phase from the workflow's mental model by documenting it as expected, or (preferred) reshape the harness so integ-runner no longer reports a spurious failure — so a reader of a run log isn't led to believe the smoke test failed when it passed.

### Use case

In run [28892156232](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/actions/runs/28892156232/job/85707381514) the log reads:

```
Verifying integration test snapshots...
NEW integ.task-api-smoke 0.002s
Tests: 1 failed, 1 total ← snapshot phase: "NEW" (no baseline)
Running integration tests for failed tests...
SUCCESS integ.task-api-smoke-TaskApiSmoke/DefaultTest 291.541s
Tests: 1 passed, 1 total ← authoritative result
```

The `1 failed` is not a test failure — it is integ-runner classifying the test as `NEW` because there is **no committed snapshot to diff against**. `/cdk/test/integ/*.snapshot/` is gitignored on purpose (each snapshot bundles ~100 MB of Lambda handler assets), and the harness runs `npx integ-runner --force` (`cdk/mise.toml:69`), which deploys-then-verifies unconditionally and ignores any snapshot verdict.

So the snapshot's normal job — an offline template diff that lets CI skip the deploy when the stack is unchanged — is structurally disabled here. It can never match, so it always reports `NEW`/failed, then always falls through to the live run. Net signal today: zero; net effect: a confusing red-looking line in every run. This repeatedly misleads anyone triaging a run (it's easy to conflate the snapshot line with the real job failure, which in that run was the separate `Ensure stack torn down` step — see #566).

Note the path filter (`^(cdk|agent)/` in `integ.yml`) gates *whether the job runs at all*, but is coarser than the snapshot (which would gate on the specific stack's synthesized template). The two are not equivalent — but given the always-`--force` design, the snapshot's finer check is thrown away regardless, which is why it can go.

### Proposed solution

Pick the lightest option that removes the misleading output:

1. **Suppress / accept the NEW-snapshot noise (minimal):** keep the current always-deploy model but add a short comment near the integ step and in `cdk/mise.toml` stating that integ-runner reports the uncommitted snapshot as `NEW`/`1 failed` on every run and the authoritative result is the live deploy-then-verify. Optionally post-process integ-runner output so the job summary shows only the live-run result.

2. **Commit a pruned, template-only snapshot (restores the offline gate):** regenerate the snapshot without bundling Lambda assets so it is small enough to commit, giving back the "skip deploy when the stack template is unchanged" optimization the snapshot exists for. More work; only worth it if we want to cut real deploys.

Recommendation: option 1 — the smoke stack is tiny and every applicable PR does a real deploy anyway, so the offline gate buys little and committing assets was already rejected as too heavy.

Out of scope: changing the deploy → assert → destroy flow itself, and the teardown/naming work in #566 / #400.

### Other information

- Related: #566 (removes the broken teardown safety net in the same workflow; the snapshot line and the teardown failure are the two separate red herrings in run 28892156232), #400 (integ harness naming/cleanup), #236 (Phase-0 parent).
- Relevant paths: `.github/workflows/integ.yml`, `cdk/mise.toml` (`[tasks.integ]`), `.gitignore` (`/cdk/test/integ/*.snapshot/`), `cdk/test/integ/integ.task-api-smoke.ts`.

Contributor guide

Open the contributing guide

Research direction

Start with cdk/mise.toml [tasks.integ] and .github/workflows/integ.yml, then inspect .gitignore and cdk/test/integ/integ.task-api-smoke.ts. Run the integ harness with its current npx integ-runner --force behavior and compare the snapshot and live-run output. Done means the run no longer presents the expected NEW snapshot as a misleading test failure, while the live deploy-and-verify result remains authoritative.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions, typescript
Domain
ci-cd, cloud, testing, tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.