monad-developers / monad-developers/ultrafuzz
Ground-truth subject binding cannot express an Ultrafuzz target distinct from the benchmark fork
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 85
- Forks
- 18
- Avg merge
- 11h 10m
- Merged PRs (30d)
- 194
Description
Scope: specific to the Aave v4 Ultrafuzz-vs-NoFuzz evaluation (#531), with a generic design
question behind it. Filed with help wanted because resolving it means deciding how much of #453's
guarantee to give up, which is a human call, not a mechanical fix.
Problem
The Aave v4 evaluation in #531 runs against clean upstream aave/aave-v4 @ 6959e3219b5506bf2acae18551cbb2a68a5b8fba
(tag v0.5.6), because the point of the experiment is that Ultrafuzz's invariant group authors its own
properties rather than reusing ScFuzzBench's hand-authored tests/recon suite.
eval score cannot score that run. The only Aave label set,
benchmarks/private-ground-truth/aave-v4-scfuzzbench/findings.yml, declares:
subject:
repository: https://github.com/scfuzzbench/aave-v4-scfuzzbench
revision: edd6c82721512540c8c90e7a36a4a8e19fd7bdf3
assertGroundTruthSubject (packages/evals/src/ground-truth.ts:168-200, called from
packages/evals/src/scoring.ts:650 and :694 whenever the target is sensitivity: private) compares
that subject against the eval row's own target.repo / target.ref and throws
EVAL_GROUND_TRUTH_SUBJECT_REPOSITORY_MISMATCH, then _REVISION_MISMATCH.
The document already knows about both commits — it records them in provenance:
provenance:
ultrafuzz_target_repo: https://github.com/aave/aave-v4
ultrafuzz_target_commit: 6959e3219b5506bf2acae18551cbb2a68a5b8fba
scfuzzbench_target_repo: https://github.com/scfuzzbench/aave-v4-scfuzzbench
scfuzzbench_target_commit: edd6c82721512540c8c90e7a36a4a8e19fd7bdf3
but only subject is validated, and subject is the benchmark fork. So the schema can express the
pairing while the validator cannot accept it.
Why the obvious workaround is not acceptable
The check only fires for sensitivity: private. Downgrading the Aave target's sensitivity would make
scoring "work" and would silently void the binding #453 added specifically so the 12 labels stay
pinned to the exact revision containing the injected harness. That must not be the resolution.
Context that matters for the decision
The ScFuzzBench Aave v4 benchmark injects no protocol mutations. It implements pre-vetted audit
contest invariants (Sherlock 1209) against the same upstream source, and the two trees differ in only
three harness-compatibility files (WETH9.sol custom errors, an AaveOracle.setSpoke test hook, and
a commented-out _disableInitializers()), with Hub.sol, Spoke.sol and every library
byte-identical. So the labelled behaviours are reachable on the upstream commit; what the fork adds is
the hand-authored oracle, which is the capability under test.
That means the pairing "labels bound to the fork, findings produced against upstream" is
scientifically sound here. The question is whether the validator should be taught to accept it, and
under what constraint.
Decision needed
- Should a ground-truth document be allowed to declare a validated Ultrafuzz-target binding
distinct from its benchmark-subject binding, such thatassertGroundTruthSubjectaccepts a row
whose target matches the former? - If yes, what keeps that from becoming a general escape hatch — for example requiring both
bindings to be full immutable SHAs, requiring an explicit equivalence attestation naming the
permitted differing paths, and recording both in scoring provenance. - If no, then #531's semantic benchmark-family credit is a documented post-hoc manual comparison
for this target, and that should be stated in the evaluation protocol rather than rediscovered
each time.
Interim position for #531, so the evaluation is not blocked: eval score is not used, the six
pre-registered metrics are assembled from typed run artifacts with per-value artifact citations, and
semantic benchmark-family credit is a documented post-hoc unblinding comparison. sensitivity is not
downgraded.
Related: #453, #506, #531.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with packages/evals/src/ground-truth.ts:168-200 and the calls from packages/evals/src/scoring.ts:650 and :694; compare those checks with the Aave ground-truth document and its provenance fields. The issue is done when the project has decided whether distinct bindings are supported, constrained the accepted data accordingly, and documented the resulting evaluation protocol or manual-comparison path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100