aws-samples / aws-samples/sample-autonomous-cloud-coding-agents
feat(ci): mutation testing on policy, validation, and orchestrator (Stryker + mutmut) (CA-04)
- Vorherrschende Sprache
- TypeScript
- Sterne
- 143
- Forks
- 46
- Ø Merge
- 3 T. 9 Std.
- Gemergte PRs (30 T.)
- 20
Beschreibung
> **This is a finding from https://github.com/krokoko/cairn** (action item **CA-04**).
### Component
Tooling / CI
### Describe the feature
Add **mutation testing** on the high-criticality modules: **Stryker** for TypeScript (`validation`, `orchestrator`) and **mutmut** for Python (`policy.py`). Run it as a **nightly** (slow-tier) job. This doubles as the AI005/AI009 gate and as the oracle-strength measurement (the "Generator + Mutant" generator-evaluator variant).
### Use case
Coverage is collected but tells us only that code *ran*, not that the tests would *catch a regression*. With no mutation testing, AI005 (tests mirroring implementation) and AI009 (happy-path-only) are essentially invisible to automation — the single highest-risk gap for a platform whose product is agents writing code. Mutation testing measures suite *adequacy* directly: a low mutation-kill score means green-but-meaningless tests on exactly the modules where correctness matters most (authorization, input validation, task orchestration).
### Proposed solution
1. Add Stryker to `cdk`, scoped to `handlers/shared/validation.ts` and the orchestrator modules.
2. Add mutmut to `agent/`, scoped to `policy.py`.
3. Run both in a scheduled nightly workflow (not on the PR critical path — it's slow).
4. Surface the mutation-kill score; set an initial target and ratchet up.
### Acceptance criteria
- [ ] Stryker configured for `validation` + `orchestrator` (TS); mutmut configured for `policy.py` (Py).
- [ ] A nightly GitHub Actions workflow runs both and reports mutation-kill scores.
- [ ] An initial mutation-kill target is recorded for each module (to be ratcheted).
- [ ] Results are captured as artifacts so the score is trackable as a per-PR/over-time delta.
### Other information
Source reports: `verification-report.md` (oracle-rot watch — "Mutation testing (absent) is the standard defense"), `verification-strategy.md` (Phase 2 #9; Generator+Mutant), `ai-smells-gates-report.md` (AI005). Effort: **M**. Depends on CA-01 (coverage floor) and CA-02 (property tests) landing first. Per ADR-003 this issue needs the `approved` label before work begins.
Beitragsleitfaden
Rechercherichtung
Beginne mit cdk/handlers/shared/validation.ts, den TypeScript-Orchestrator-Modulen und agent/policy.py. Überprüfe anschließend die bestehende GitHub-Actions-Konfiguration und die referenzierten Verifizierungsberichte. Konfiguriere Stryker und mutmut für die genannten Module, führe sie in einem nächtlichen Slow-Tier-Workflow aus und stelle sicher, dass Mutation-Kill-Scores, anfängliche Ziele und Artefakte zur zeitlichen Nachverfolgung berichtet werden.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- github-actions, python, typescript
- Bereich
- ci-cd, testing, tooling
- Issue-Typ
- Feature
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 48/100