aws-samples / aws-samples/sample-autonomous-cloud-coding-agents
feat(cdk): CDK integ-tests for deployed runtime E2E verification
- Dominant language
- TypeScript
- Stars
- 143
- Forks
- 46
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 20
Description
### Component
CDK / infrastructure, API or orchestration, Tooling / CI
### Describe the feature
Introduce **automated, deploy-then-verify integration tests** for the ABCA platform using the experimental [`@aws-cdk/integ-tests-alpha`](https://docs.aws.amazon.com/cdk/api/v2/docs/integ-tests-alpha-readme.html) library and the **integ-runner** workflow.
Today we have strong unit and handler-level coverage, but full-stack **runtime behavior** is validated manually (for example Cedar HITL scenarios A–E against a live `backgroundagent-dev` stack, as described in `docs/design/CEDAR_HITL_GATES.md`). ADR-008 Level 3 calls for integration or E2E coverage on critical paths; this issue tracks making that repeatable in CI/CD: **synthesize → deploy → assert → destroy**, with no long-lived drift in shared accounts.
Each integ test should be a **1:1 CDK application** (per AWS guidance): an `IntegTest` construct registering one or more stack test cases, plus optional `DeployAssert` assertions (`awsApiCall`, `httpApiCall`, `invokeFunction`, `waitForAssertions`) to poll until terminal conditions are met.
### Use case
- **Confidence before merge:** Catch regressions in orchestration, admission, hydration, agent session lifecycle, and channel hand-offs that mocks cannot surface.
- **Safer iteration on roadmap priorities:** Identity propagation, cost attribution, registry resolution, and Linear workflows all need a trustworthy “does the deployed stack actually work?” gate.
- **Replace ad-hoc smoke tests:** Operators should not be the only path to validate a post-deploy stack; failures should surface in CI with clear assertion output.
- **Document the contract:** Assertions encode expected task states, event shapes, and API responses—the living spec for platform behavior.
### Proposed solution
**Phase 0 — Foundation**
- Add `@aws-cdk/integ-tests-alpha` to `cdk/` (experimental; pin version; document upgrade risk in CONTRIBUTING or a short design note).
- Add `cdk/test/integ/` (or `cdk/integ/`) with at least one **smoke integ** that deploys the main stack (or a trimmed integ variant if full stack is too heavy) and asserts a minimal happy path, e.g.:
- `POST /v1/tasks` (or internal test hook) creates a task
- Poll DynamoDB / `GET /v1/tasks/{id}` until a terminal state
- Verify required fields on the task record (`task_id`, `user_id`, status, timestamps)
- Wire **mise** / CI entry point (e.g. `mise //cdk:integ`) invoking `integ-runner` with documented prerequisites (AWS credentials, target account/region, optional dedicated integ stack name).
**Phase 1 — Core lifecycle**
- Scenarios aligned with manual Cedar HITL E2E matrix where feasible (submit → run → complete / fail / await approval → approve or deny → terminal).
- Use `waitForAssertions()` for long-running agent/orchestrator paths with conservative timeouts.
- Ensure **teardown** (`destroy` with force) runs on success and failure; document cost and account isolation expectations.
**Phase 2 — Channels and guardrails** (follow-up issues OK)
- Webhook create-task, Slack/Linear adapters (may require test fixtures or mocked externals).
- Stranded-task reconciler behavior under controlled clock/fixtures if possible.
**Design constraints**
- Integ tests must **not** run on every PR by default if cost/latency is prohibitive—options: nightly workflow, `workflow_dispatch`, or label-gated job (document the chosen policy in the issue implementation PR).
- Prefer a **dedicated integ stack name** or ephemeral account to avoid colliding with developer `backgroundagent-dev` stacks.
- Keep integ apps **separate** from production synth (`cdk.out` isolation); assertion stacks use `DeployAssert` per library docs.
**Out of scope (for this issue)**
- Replacing unit tests or moto-backed handler tests (see #61 for trace round-trip).
- Full agent “open PR on real GitHub repo” unless we add a gated fixture repo and PAT—can be a child issue.
### Other information
- **Roadmap gap:** “Deployed runtime E2E verification” is a prioritized platform item but not yet listed explicitly in `docs/guides/ROADMAP.md`—add a row when implementation starts (source + Starlight sync).
- **Related:** ADR-008 Level 3; `docs/design/CEDAR_HITL_GATES.md` §15.3 (manual E2E on `backgroundagent-dev`); ADR-013 Tier 3 (remote CI as authoritative, integ as pre-merge signal).
- **Reference:** [@aws-cdk/integ-tests-alpha README](https://docs.aws.amazon.com/cdk/api/v2/docs/integ-tests-alpha-readme.html)
### Acknowledgements
- [ ] I may be able to implement this feature
- [ ] This might be a breaking change
Contributor guide
Assessment
This issue has not been assessed yet.