aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

feat(cdk): CDK integ-tests for deployed runtime E2E verification

Open
#236 4 comments 0 reactions 1 assignee Claimed by @ayushtr-aws View on GitHub
approved enhancement infra-cdk P0 validation-loop
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

### Component

CDK / infrastructure, API or orchestration, Tooling / CI

### Describe the feature

Introduce **automated, deploy-then-verify integration tests** for the ABCA platform using the experimental [`@aws-cdk/integ-tests-alpha`](https://docs.aws.amazon.com/cdk/api/v2/docs/integ-tests-alpha-readme.html) library and the **integ-runner** workflow.

Today we have strong unit and handler-level coverage, but full-stack **runtime behavior** is validated manually (for example Cedar HITL scenarios A–E against a live `backgroundagent-dev` stack, as described in `docs/design/CEDAR_HITL_GATES.md`). ADR-008 Level 3 calls for integration or E2E coverage on critical paths; this issue tracks making that repeatable in CI/CD: **synthesize → deploy → assert → destroy**, with no long-lived drift in shared accounts.

Each integ test should be a **1:1 CDK application** (per AWS guidance): an `IntegTest` construct registering one or more stack test cases, plus optional `DeployAssert` assertions (`awsApiCall`, `httpApiCall`, `invokeFunction`, `waitForAssertions`) to poll until terminal conditions are met.

### Use case

- **Confidence before merge:** Catch regressions in orchestration, admission, hydration, agent session lifecycle, and channel hand-offs that mocks cannot surface.
- **Safer iteration on roadmap priorities:** Identity propagation, cost attribution, registry resolution, and Linear workflows all need a trustworthy “does the deployed stack actually work?” gate.
- **Replace ad-hoc smoke tests:** Operators should not be the only path to validate a post-deploy stack; failures should surface in CI with clear assertion output.
- **Document the contract:** Assertions encode expected task states, event shapes, and API responses—the living spec for platform behavior.

### Proposed solution

**Phase 0 — Foundation**
- Add `@aws-cdk/integ-tests-alpha` to `cdk/` (experimental; pin version; document upgrade risk in CONTRIBUTING or a short design note).
- Add `cdk/test/integ/` (or `cdk/integ/`) with at least one **smoke integ** that deploys the main stack (or a trimmed integ variant if full stack is too heavy) and asserts a minimal happy path, e.g.:
- `POST /v1/tasks` (or internal test hook) creates a task
- Poll DynamoDB / `GET /v1/tasks/{id}` until a terminal state
- Verify required fields on the task record (`task_id`, `user_id`, status, timestamps)
- Wire **mise** / CI entry point (e.g. `mise //cdk:integ`) invoking `integ-runner` with documented prerequisites (AWS credentials, target account/region, optional dedicated integ stack name).

**Phase 1 — Core lifecycle**
- Scenarios aligned with manual Cedar HITL E2E matrix where feasible (submit → run → complete / fail / await approval → approve or deny → terminal).
- Use `waitForAssertions()` for long-running agent/orchestrator paths with conservative timeouts.
- Ensure **teardown** (`destroy` with force) runs on success and failure; document cost and account isolation expectations.

**Phase 2 — Channels and guardrails** (follow-up issues OK)
- Webhook create-task, Slack/Linear adapters (may require test fixtures or mocked externals).
- Stranded-task reconciler behavior under controlled clock/fixtures if possible.

**Design constraints**
- Integ tests must **not** run on every PR by default if cost/latency is prohibitive—options: nightly workflow, `workflow_dispatch`, or label-gated job (document the chosen policy in the issue implementation PR).
- Prefer a **dedicated integ stack name** or ephemeral account to avoid colliding with developer `backgroundagent-dev` stacks.
- Keep integ apps **separate** from production synth (`cdk.out` isolation); assertion stacks use `DeployAssert` per library docs.

**Out of scope (for this issue)**
- Replacing unit tests or moto-backed handler tests (see #61 for trace round-trip).
- Full agent “open PR on real GitHub repo” unless we add a gated fixture repo and PAT—can be a child issue.

### Other information

- **Roadmap gap:** “Deployed runtime E2E verification” is a prioritized platform item but not yet listed explicitly in `docs/guides/ROADMAP.md`—add a row when implementation starts (source + Starlight sync).
- **Related:** ADR-008 Level 3; `docs/design/CEDAR_HITL_GATES.md` §15.3 (manual E2E on `backgroundagent-dev`); ADR-013 Tier 3 (remote CI as authoritative, integ as pre-merge signal).
- **Reference:** [@aws-cdk/integ-tests-alpha README](https://docs.aws.amazon.com/cdk/api/v2/docs/integ-tests-alpha-readme.html)

### Acknowledgements

- [ ] I may be able to implement this feature
- [ ] This might be a breaking change

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.