aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

feat(cdk): stop integ harness from surfacing a misleading NEW-snapshot failure

Abierto
#569 0 comentarios 0 reacciones 0 asignados Ver en GitHub
ci-cd enhancement infra-cdk tooling
Lenguaje dominante
TypeScript
Estrellas
143
Forks
46
Merge medio
3 d 10 h
PR fusionados (30 d)
24

Descripción

### Component

CDK / infrastructure (Tooling / CI — integ harness)

### Describe the feature

Stop the integ smoke harness from carrying an integ-runner **snapshot** that provides no signal and surfaces a misleading `1 failed` line on every run. Either remove the snapshot phase from the workflow's mental model by documenting it as expected, or (preferred) reshape the harness so integ-runner no longer reports a spurious failure — so a reader of a run log isn't led to believe the smoke test failed when it passed.

### Use case

In run [28892156232](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/actions/runs/28892156232/job/85707381514) the log reads:

```
Verifying integration test snapshots...
NEW integ.task-api-smoke 0.002s
Tests: 1 failed, 1 total ← snapshot phase: "NEW" (no baseline)
Running integration tests for failed tests...
SUCCESS integ.task-api-smoke-TaskApiSmoke/DefaultTest 291.541s
Tests: 1 passed, 1 total ← authoritative result
```

The `1 failed` is not a test failure — it is integ-runner classifying the test as `NEW` because there is **no committed snapshot to diff against**. `/cdk/test/integ/*.snapshot/` is gitignored on purpose (each snapshot bundles ~100 MB of Lambda handler assets), and the harness runs `npx integ-runner --force` (`cdk/mise.toml:69`), which deploys-then-verifies unconditionally and ignores any snapshot verdict.

So the snapshot's normal job — an offline template diff that lets CI skip the deploy when the stack is unchanged — is structurally disabled here. It can never match, so it always reports `NEW`/failed, then always falls through to the live run. Net signal today: zero; net effect: a confusing red-looking line in every run. This repeatedly misleads anyone triaging a run (it's easy to conflate the snapshot line with the real job failure, which in that run was the separate `Ensure stack torn down` step — see #566).

Note the path filter (`^(cdk|agent)/` in `integ.yml`) gates *whether the job runs at all*, but is coarser than the snapshot (which would gate on the specific stack's synthesized template). The two are not equivalent — but given the always-`--force` design, the snapshot's finer check is thrown away regardless, which is why it can go.

### Proposed solution

Pick the lightest option that removes the misleading output:

1. **Suppress / accept the NEW-snapshot noise (minimal):** keep the current always-deploy model but add a short comment near the integ step and in `cdk/mise.toml` stating that integ-runner reports the uncommitted snapshot as `NEW`/`1 failed` on every run and the authoritative result is the live deploy-then-verify. Optionally post-process integ-runner output so the job summary shows only the live-run result.

2. **Commit a pruned, template-only snapshot (restores the offline gate):** regenerate the snapshot without bundling Lambda assets so it is small enough to commit, giving back the "skip deploy when the stack template is unchanged" optimization the snapshot exists for. More work; only worth it if we want to cut real deploys.

Recommendation: option 1 — the smoke stack is tiny and every applicable PR does a real deploy anyway, so the offline gate buys little and committing assets was already rejected as too heavy.

Out of scope: changing the deploy → assert → destroy flow itself, and the teardown/naming work in #566 / #400.

### Other information

- Related: #566 (removes the broken teardown safety net in the same workflow; the snapshot line and the teardown failure are the two separate red herrings in run 28892156232), #400 (integ harness naming/cleanup), #236 (Phase-0 parent).
- Relevant paths: `.github/workflows/integ.yml`, `cdk/mise.toml` (`[tasks.integ]`), `.gitignore` (`/cdk/test/integ/*.snapshot/`), `cdk/test/integ/integ.task-api-smoke.ts`.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Empieza por cdk/mise.toml [tasks.integ] y .github/workflows/integ.yml; después inspecciona .gitignore y cdk/test/integ/integ.task-api-smoke.ts. Ejecuta el integ harness con su comportamiento actual de npx integ-runner --force y compara la salida del snapshot con la de la ejecución en vivo. Se considera terminado cuando la ejecución ya no presenta el snapshot NEW esperado como un fallo de prueba engañoso, mientras que el resultado del despliegue y la verificación en vivo sigue siendo la autoridad.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
aws, github-actions, typescript
Área
ci-cd, cloud, testing, tooling
Tipo de issue
Nueva funcionalidad
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
55/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.