aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

feat(cdk): CDK integ-tests for deployed runtime E2E verification

Abierto
#236 4 comentarios 0 reacciones 1 asignado Reclamado por @ayushtr-aws Ver en GitHub
approved enhancement infra-cdk P0 validation-loop
Lenguaje dominante
TypeScript
Estrellas
143
Forks
46
Merge medio
3 d 10 h
PR fusionados (30 d)
24

Descripción

### Component

CDK / infrastructure, API or orchestration, Tooling / CI

### Describe the feature

Introduce **automated, deploy-then-verify integration tests** for the ABCA platform using the experimental [`@aws-cdk/integ-tests-alpha`](https://docs.aws.amazon.com/cdk/api/v2/docs/integ-tests-alpha-readme.html) library and the **integ-runner** workflow.

Today we have strong unit and handler-level coverage, but full-stack **runtime behavior** is validated manually (for example Cedar HITL scenarios A–E against a live `backgroundagent-dev` stack, as described in `docs/design/CEDAR_HITL_GATES.md`). ADR-008 Level 3 calls for integration or E2E coverage on critical paths; this issue tracks making that repeatable in CI/CD: **synthesize → deploy → assert → destroy**, with no long-lived drift in shared accounts.

Each integ test should be a **1:1 CDK application** (per AWS guidance): an `IntegTest` construct registering one or more stack test cases, plus optional `DeployAssert` assertions (`awsApiCall`, `httpApiCall`, `invokeFunction`, `waitForAssertions`) to poll until terminal conditions are met.

### Use case

- **Confidence before merge:** Catch regressions in orchestration, admission, hydration, agent session lifecycle, and channel hand-offs that mocks cannot surface.
- **Safer iteration on roadmap priorities:** Identity propagation, cost attribution, registry resolution, and Linear workflows all need a trustworthy “does the deployed stack actually work?” gate.
- **Replace ad-hoc smoke tests:** Operators should not be the only path to validate a post-deploy stack; failures should surface in CI with clear assertion output.
- **Document the contract:** Assertions encode expected task states, event shapes, and API responses—the living spec for platform behavior.

### Proposed solution

**Phase 0 — Foundation**
- Add `@aws-cdk/integ-tests-alpha` to `cdk/` (experimental; pin version; document upgrade risk in CONTRIBUTING or a short design note).
- Add `cdk/test/integ/` (or `cdk/integ/`) with at least one **smoke integ** that deploys the main stack (or a trimmed integ variant if full stack is too heavy) and asserts a minimal happy path, e.g.:
- `POST /v1/tasks` (or internal test hook) creates a task
- Poll DynamoDB / `GET /v1/tasks/{id}` until a terminal state
- Verify required fields on the task record (`task_id`, `user_id`, status, timestamps)
- Wire **mise** / CI entry point (e.g. `mise //cdk:integ`) invoking `integ-runner` with documented prerequisites (AWS credentials, target account/region, optional dedicated integ stack name).

**Phase 1 — Core lifecycle**
- Scenarios aligned with manual Cedar HITL E2E matrix where feasible (submit → run → complete / fail / await approval → approve or deny → terminal).
- Use `waitForAssertions()` for long-running agent/orchestrator paths with conservative timeouts.
- Ensure **teardown** (`destroy` with force) runs on success and failure; document cost and account isolation expectations.

**Phase 2 — Channels and guardrails** (follow-up issues OK)
- Webhook create-task, Slack/Linear adapters (may require test fixtures or mocked externals).
- Stranded-task reconciler behavior under controlled clock/fixtures if possible.

**Design constraints**
- Integ tests must **not** run on every PR by default if cost/latency is prohibitive—options: nightly workflow, `workflow_dispatch`, or label-gated job (document the chosen policy in the issue implementation PR).
- Prefer a **dedicated integ stack name** or ephemeral account to avoid colliding with developer `backgroundagent-dev` stacks.
- Keep integ apps **separate** from production synth (`cdk.out` isolation); assertion stacks use `DeployAssert` per library docs.

**Out of scope (for this issue)**
- Replacing unit tests or moto-backed handler tests (see #61 for trace round-trip).
- Full agent “open PR on real GitHub repo” unless we add a gated fixture repo and PAT—can be a child issue.

### Other information

- **Roadmap gap:** “Deployed runtime E2E verification” is a prioritized platform item but not yet listed explicitly in `docs/guides/ROADMAP.md`—add a row when implementation starts (source + Starlight sync).
- **Related:** ADR-008 Level 3; `docs/design/CEDAR_HITL_GATES.md` §15.3 (manual E2E on `backgroundagent-dev`); ADR-013 Tier 3 (remote CI as authoritative, integ as pre-merge signal).
- **Reference:** [@aws-cdk/integ-tests-alpha README](https://docs.aws.amazon.com/cdk/api/v2/docs/integ-tests-alpha-readme.html)

### Acknowledgements

- [ ] I may be able to implement this feature
- [ ] This might be a breaking change

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.