monad-developers / monad-developers/ultrafuzz

Ground-truth subject binding cannot express an Ultrafuzz target distinct from the benchmark fork

Open
#544 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

deferred-post-v0.1 help wanted
Dominant language
TypeScript
Stars
85
Forks
18
Avg merge
11h 10m
Merged PRs (30d)
194

Description

Scope: specific to the Aave v4 Ultrafuzz-vs-NoFuzz evaluation (#531), with a generic design
question behind it. Filed with help wanted because resolving it means deciding how much of #453's
guarantee to give up, which is a human call, not a mechanical fix.

Problem

The Aave v4 evaluation in #531 runs against clean upstream aave/aave-v4 @ 6959e3219b5506bf2acae18551cbb2a68a5b8fba
(tag v0.5.6), because the point of the experiment is that Ultrafuzz's invariant group authors its own
properties rather than reusing ScFuzzBench's hand-authored tests/recon suite.

eval score cannot score that run. The only Aave label set,
benchmarks/private-ground-truth/aave-v4-scfuzzbench/findings.yml, declares:

subject:
  repository: https://github.com/scfuzzbench/aave-v4-scfuzzbench
  revision: edd6c82721512540c8c90e7a36a4a8e19fd7bdf3

assertGroundTruthSubject (packages/evals/src/ground-truth.ts:168-200, called from
packages/evals/src/scoring.ts:650 and :694 whenever the target is sensitivity: private) compares
that subject against the eval row's own target.repo / target.ref and throws
EVAL_GROUND_TRUTH_SUBJECT_REPOSITORY_MISMATCH, then _REVISION_MISMATCH.

The document already knows about both commits — it records them in provenance:

provenance:
  ultrafuzz_target_repo: https://github.com/aave/aave-v4
  ultrafuzz_target_commit: 6959e3219b5506bf2acae18551cbb2a68a5b8fba
  scfuzzbench_target_repo: https://github.com/scfuzzbench/aave-v4-scfuzzbench
  scfuzzbench_target_commit: edd6c82721512540c8c90e7a36a4a8e19fd7bdf3

but only subject is validated, and subject is the benchmark fork. So the schema can express the
pairing while the validator cannot accept it.

Why the obvious workaround is not acceptable

The check only fires for sensitivity: private. Downgrading the Aave target's sensitivity would make
scoring "work" and would silently void the binding #453 added specifically so the 12 labels stay
pinned to the exact revision containing the injected harness. That must not be the resolution.

Context that matters for the decision

The ScFuzzBench Aave v4 benchmark injects no protocol mutations. It implements pre-vetted audit
contest invariants (Sherlock 1209) against the same upstream source, and the two trees differ in only
three harness-compatibility files (WETH9.sol custom errors, an AaveOracle.setSpoke test hook, and
a commented-out _disableInitializers()), with Hub.sol, Spoke.sol and every library
byte-identical. So the labelled behaviours are reachable on the upstream commit; what the fork adds is
the hand-authored oracle, which is the capability under test.

That means the pairing "labels bound to the fork, findings produced against upstream" is
scientifically sound here. The question is whether the validator should be taught to accept it, and
under what constraint.

Decision needed

  • Should a ground-truth document be allowed to declare a validated Ultrafuzz-target binding
    distinct from its benchmark-subject binding, such that assertGroundTruthSubject accepts a row
    whose target matches the former?
  • If yes, what keeps that from becoming a general escape hatch — for example requiring both
    bindings to be full immutable SHAs, requiring an explicit equivalence attestation naming the
    permitted differing paths, and recording both in scoring provenance.
  • If no, then #531's semantic benchmark-family credit is a documented post-hoc manual comparison
    for this target, and that should be stated in the evaluation protocol rather than rediscovered
    each time.

Interim position for #531, so the evaluation is not blocked: eval score is not used, the six
pre-registered metrics are assembled from typed run artifacts with per-value artifact citations, and
semantic benchmark-family credit is a documented post-hoc unblinding comparison. sensitivity is not
downgraded.

Related: #453, #506, #531.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/evals/src/ground-truth.ts:168-200 and the calls from packages/evals/src/scoring.ts:650 and :694; compare those checks with the Aave ground-truth document and its provenance fields. The issue is done when the project has decided whether distinct bindings are supported, constrained the accepted data accordingly, and documented the resulting evaluation protocol or manual-comparison path.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.