aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

feat(testing): property-based tests for validation, numeric utils, and Cedar determinism (CA-02)

Ouverte
#253 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
validation-loop
Langage dominant
TypeScript
Étoiles
143
Forks
46
Merge moyen
3 j 10 h
PR mergées (30 j)
24

Description

> **This is a finding from https://github.com/krokoko/cairn** (action item **CA-02**).

### Component

API or orchestration / Agent (Python runtime)

### Describe the feature

Add **property-based tests** (`fast-check` for TypeScript, `Hypothesis` for Python) for the exactly-verifiable surfaces of the codebase:

- **Validation / bounds** (`cdk/src/handlers/shared/validation.ts`, `numeric.ts`) — generators **driven from `contracts/constants.json`** so the oracle cannot rot when bounds change. Assert: in-range values accepted, out-of-range rejected, normalization is idempotent (`normalize(normalize(x)) == normalize(x)`), parse↔serialize round-trips, monotonic clamping.
- **Numeric / timestamp utils** — round-trip and monotonicity properties.
- **Cedar decision determinism** — decision invariant under attribute reordering; deny overrides permit; unknown action → default deny (complements the existing `contracts/cedar-parity/` differential golden suite).

Mirror the validation properties in Python with Hypothesis where the agent re-validates input.

### Use case

The validation/bounds and numeric suites today assert **hardcoded expected values** duplicated from `contracts/constants.json`. This is an oracle-rot risk: when a bound changes in `constants.json`, the hardcoded expectations silently drift and the suite proves nothing. Property-based testing replaces brittle example-based oracles with invariants derived from the single source of truth, and is the cheapest high-leverage oracle for the deterministic, "exactly-verifiable" surfaces. This also closes AI009 (happy-path-only coverage) on these surfaces.

### Proposed solution

1. Add `fast-check` as a dev dependency to `cdk` (and `cli` where `format.ts` warrants it).
2. For each bounded field, load the bound from `contracts/constants.json` and generate values inside/outside the range, asserting accept/reject.
3. Assert normalization idempotence and parse↔serialize round-trips.
4. Add `Hypothesis` to `agent/` and mirror the validation properties where the agent re-validates.
5. Add a permutation property for Cedar decision determinism alongside the existing parity fixtures.

### Acceptance criteria

- [ ] `fast-check` (TS) and `Hypothesis` (Py) are wired into the existing test tasks (`mise //cdk:test`, `mise //agent:test`).
- [ ] Validation/bounds generators read ranges from `contracts/constants.json` — **no** hardcoded bound literals in the property tests.
- [ ] Properties cover: in-range accept, out-of-range reject, idempotent normalize, round-trip serialize.
- [ ] Cedar decision-determinism property added (attribute reordering invariant; unknown action → deny).
- [ ] Tests run in the standard unit lane and pass in CI's required `build` check.

### Other information

Source reports: `readiness-roadmap.md`, `verification-report.md` (Missing Oracles / oracle-rot watch), `verification-strategy.md` (§"Validation / bounds (highest impact-per-effort)"), `ai-smells-gates-report.md` (AI009). Effort: **M**. Depends on CA-01 (coverage floor) landing first. Per ADR-003 this issue needs the `approved` label before work begins.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par cdk/src/handlers/shared/validation.ts, numeric.ts, contracts/constants.json et les fixtures existants de contracts/cedar-parity ; exécutez mise //cdk:test et mise //agent:test pour comprendre les lanes de tests unitaires actuels. Inspectez ensuite les chemins de validation de l’agent et les tests Cedar existants. C’est terminé lorsque les propriétés TypeScript et Python utilisent des bornes dérivées du contrat, que le déterminisme de Cedar est couvert et que les deux tâches de test standard passent la vérification de build requise.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python, typescript
Domaine
backend-api-design, testing
Type d'issue
Fonctionnalité
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Calme
Clarté
Plutôt claire
Accessibilité débutants
42/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.