aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

feat(testing): property-based tests for validation, numeric utils, and Cedar determinism (CA-02)

Offen
#253 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
validation-loop
Vorherrschende Sprache
TypeScript
Sterne
143
Forks
46
Ø Merge
3 T. 10 Std.
Gemergte PRs (30 T.)
24

Beschreibung

> **This is a finding from https://github.com/krokoko/cairn** (action item **CA-02**).

### Component

API or orchestration / Agent (Python runtime)

### Describe the feature

Add **property-based tests** (`fast-check` for TypeScript, `Hypothesis` for Python) for the exactly-verifiable surfaces of the codebase:

- **Validation / bounds** (`cdk/src/handlers/shared/validation.ts`, `numeric.ts`) — generators **driven from `contracts/constants.json`** so the oracle cannot rot when bounds change. Assert: in-range values accepted, out-of-range rejected, normalization is idempotent (`normalize(normalize(x)) == normalize(x)`), parse↔serialize round-trips, monotonic clamping.
- **Numeric / timestamp utils** — round-trip and monotonicity properties.
- **Cedar decision determinism** — decision invariant under attribute reordering; deny overrides permit; unknown action → default deny (complements the existing `contracts/cedar-parity/` differential golden suite).

Mirror the validation properties in Python with Hypothesis where the agent re-validates input.

### Use case

The validation/bounds and numeric suites today assert **hardcoded expected values** duplicated from `contracts/constants.json`. This is an oracle-rot risk: when a bound changes in `constants.json`, the hardcoded expectations silently drift and the suite proves nothing. Property-based testing replaces brittle example-based oracles with invariants derived from the single source of truth, and is the cheapest high-leverage oracle for the deterministic, "exactly-verifiable" surfaces. This also closes AI009 (happy-path-only coverage) on these surfaces.

### Proposed solution

1. Add `fast-check` as a dev dependency to `cdk` (and `cli` where `format.ts` warrants it).
2. For each bounded field, load the bound from `contracts/constants.json` and generate values inside/outside the range, asserting accept/reject.
3. Assert normalization idempotence and parse↔serialize round-trips.
4. Add `Hypothesis` to `agent/` and mirror the validation properties where the agent re-validates.
5. Add a permutation property for Cedar decision determinism alongside the existing parity fixtures.

### Acceptance criteria

- [ ] `fast-check` (TS) and `Hypothesis` (Py) are wired into the existing test tasks (`mise //cdk:test`, `mise //agent:test`).
- [ ] Validation/bounds generators read ranges from `contracts/constants.json` — **no** hardcoded bound literals in the property tests.
- [ ] Properties cover: in-range accept, out-of-range reject, idempotent normalize, round-trip serialize.
- [ ] Cedar decision-determinism property added (attribute reordering invariant; unknown action → deny).
- [ ] Tests run in the standard unit lane and pass in CI's required `build` check.

### Other information

Source reports: `readiness-roadmap.md`, `verification-report.md` (Missing Oracles / oracle-rot watch), `verification-strategy.md` (§"Validation / bounds (highest impact-per-effort)"), `ai-smells-gates-report.md` (AI009). Effort: **M**. Depends on CA-01 (coverage floor) landing first. Per ADR-003 this issue needs the `approved` label before work begins.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne mit cdk/src/handlers/shared/validation.ts, numeric.ts, contracts/constants.json und den vorhandenen contracts/cedar-parity/-Fixtures; führe mise //cdk:test und mise //agent:test aus, um die aktuellen Unit-Test-Lanes zu verstehen. Untersuche anschließend die Validierungspfade des Agents und die vorhandenen Cedar-Tests. Die Aufgabe ist erledigt, wenn die TypeScript- und Python-Eigenschaften aus dem Contract abgeleitete Grenzen verwenden, der Determinismus von Cedar abgedeckt ist und beide Standard-Testaufgaben die erforderliche Build-Prüfung bestehen.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, typescript
Bereich
backend-api-design, testing
Issue-Typ
Feature
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
42/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.