monad-developers / monad-developers/ultrafuzz
Hill-climb property generation quality
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 85
- Forks
- 18
- Avg merge
- 11h 10m
- Merged PRs (30d)
- 194
Description
Goal
Improve property-generation prompts so they produce more valid, interesting properties with fewer property-level false positives.
This is a follow-up to #79. Property provenance must land first so outcomes can be attributed to the exact source lens/framework and generated property.
Definitions
- Valid property: consistent with the target protocol's intended behavior, reachable, non-vacuous, and expressible as an observable oracle.
- Interesting property: project-specific and capable of detecting a meaningful state violation rather than restating a generic or duplicate assertion.
- Property false positive: a generated property that would flag intended protocol behavior as a violation.
V1 scope
- Define a structured adjudication record keyed by the canonical property ID from #79:
property_idvalid | invalid | needs-reviewinteresting: booleanfalse_positive: booleanrationale
- Aggregate valid rate, interesting rate, and false-positive rate by source lens/framework.
- Define a deterministic score for comparing a candidate property-generation prompt against its baseline.
- Run candidates against a fixed, pinned benchmark cohort for multiple trials.
- Accept a candidate only when it improves the aggregate score without increasing the false-positive rate; otherwise retain the baseline.
- Preserve the candidate prompt/config, benchmark lineage, per-property adjudications, aggregate score, and accept/reject decision as artifacts.
Non-goals
- Property provenance itself; that belongs to #79.
- Changing invariant fuzzer selection or scheduling.
- Automatically changing target production code.
- Optimizing downstream finding triage.
Acceptance criteria
- A fixed benchmark cohort and trial count can be configured.
- Baseline and candidate runs use compatible benchmark and scoring lineage.
- Each adjudicated property links to its canonical property ID and source provenance from #79.
- Reports show valid, interesting, and false-positive rates overall and per source lens/framework.
- Candidate acceptance is deterministic from the recorded inputs and scoring rule.
- Rejected candidates do not replace the active property-generation prompt.
- Tests cover improved, regressed, tied, and incomplete-adjudication outcomes.
- Documentation explains the scoring rule, artifacts, and how to reproduce an optimization run.
Dependency
Blocked by #79.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the dependency on issue #79, since property provenance and canonical property IDs must be available first. Use the V1 scope and acceptance criteria to define the benchmark, adjudication, scoring, artifact, and decision requirements; done means the listed improved, regressed, tied, and incomplete-adjudication outcomes are tested and the optimization process is reproducible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100