aws-samples / aws-samples/sample-autonomous-cloud-coding-agents
feat(testing): property-based tests for validation, numeric utils, and Cedar determinism (CA-02)
- Langage dominant
- TypeScript
- Étoiles
- 143
- Forks
- 46
- Merge moyen
- 3 j 10 h
- PR mergées (30 j)
- 24
Description
> **This is a finding from https://github.com/krokoko/cairn** (action item **CA-02**).
### Component
API or orchestration / Agent (Python runtime)
### Describe the feature
Add **property-based tests** (`fast-check` for TypeScript, `Hypothesis` for Python) for the exactly-verifiable surfaces of the codebase:
- **Validation / bounds** (`cdk/src/handlers/shared/validation.ts`, `numeric.ts`) — generators **driven from `contracts/constants.json`** so the oracle cannot rot when bounds change. Assert: in-range values accepted, out-of-range rejected, normalization is idempotent (`normalize(normalize(x)) == normalize(x)`), parse↔serialize round-trips, monotonic clamping.
- **Numeric / timestamp utils** — round-trip and monotonicity properties.
- **Cedar decision determinism** — decision invariant under attribute reordering; deny overrides permit; unknown action → default deny (complements the existing `contracts/cedar-parity/` differential golden suite).
Mirror the validation properties in Python with Hypothesis where the agent re-validates input.
### Use case
The validation/bounds and numeric suites today assert **hardcoded expected values** duplicated from `contracts/constants.json`. This is an oracle-rot risk: when a bound changes in `constants.json`, the hardcoded expectations silently drift and the suite proves nothing. Property-based testing replaces brittle example-based oracles with invariants derived from the single source of truth, and is the cheapest high-leverage oracle for the deterministic, "exactly-verifiable" surfaces. This also closes AI009 (happy-path-only coverage) on these surfaces.
### Proposed solution
1. Add `fast-check` as a dev dependency to `cdk` (and `cli` where `format.ts` warrants it).
2. For each bounded field, load the bound from `contracts/constants.json` and generate values inside/outside the range, asserting accept/reject.
3. Assert normalization idempotence and parse↔serialize round-trips.
4. Add `Hypothesis` to `agent/` and mirror the validation properties where the agent re-validates.
5. Add a permutation property for Cedar decision determinism alongside the existing parity fixtures.
### Acceptance criteria
- [ ] `fast-check` (TS) and `Hypothesis` (Py) are wired into the existing test tasks (`mise //cdk:test`, `mise //agent:test`).
- [ ] Validation/bounds generators read ranges from `contracts/constants.json` — **no** hardcoded bound literals in the property tests.
- [ ] Properties cover: in-range accept, out-of-range reject, idempotent normalize, round-trip serialize.
- [ ] Cedar decision-determinism property added (attribute reordering invariant; unknown action → deny).
- [ ] Tests run in the standard unit lane and pass in CI's required `build` check.
### Other information
Source reports: `readiness-roadmap.md`, `verification-report.md` (Missing Oracles / oracle-rot watch), `verification-strategy.md` (§"Validation / bounds (highest impact-per-effort)"), `ai-smells-gates-report.md` (AI009). Effort: **M**. Depends on CA-01 (coverage floor) landing first. Per ADR-003 this issue needs the `approved` label before work begins.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par cdk/src/handlers/shared/validation.ts, numeric.ts, contracts/constants.json et les fixtures existants de contracts/cedar-parity ; exécutez mise //cdk:test et mise //agent:test pour comprendre les lanes de tests unitaires actuels. Inspectez ensuite les chemins de validation de l’agent et les tests Cedar existants. C’est terminé lorsque les propriétés TypeScript et Python utilisent des bornes dérivées du contrat, que le déterminisme de Cedar est couvert et que les deux tâches de test standard passent la vérification de build requise.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python, typescript
- Domaine
- backend-api-design, testing
- Type d'issue
- Fonctionnalité
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Calme
- Clarté
- Plutôt claire
- Accessibilité débutants
- 42/100