zivtech / zivtech/react-critic

fix: benchmark reports score static coverage maps, not model outputs

Open
#1 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
0
Avg merge
59m
Merged PRs (30d)
2

Description

Problem

At commit 5ed06f6e2667835912dc1b6cf1c40cc7da6dd07c, scripts/run_benchmark.py does not invoke a candidate or baseline model and does not read captured model responses. run_seed() calls score_fixture() twice against the static CRITIC_COVERAGE and BASELINE_COVERAGE dictionaries, then publishes those derived values as harsh/baseline scores in the three benchmark reports.

The external-skill evidence audit therefore retains the React, Next.js, and React Native result sets as rejected-integrity. They are rubric-coverage projections, not evidence of model behavior.

Required repair

  • Make benchmark scoring consume immutable captured candidate and baseline outputs, with provider/model/parameters and input/output hashes.
  • Fail closed when either response, its provenance, or the scorer/rubric identity is missing.
  • Keep static rubric-coverage analysis separate and label it as static analysis if it remains useful.
  • Add tests proving that changing a captured response can change the score and that missing responses cannot produce a benchmark report.
  • Emit a provenance-complete run manifest for each scored cell.

Acceptance boundary

Repairing the runner does not rehabilitate the existing reports. They remain rejected until a separately approved run produces complete captured outputs and manifests. This issue does not authorize paid model calls.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in scripts/run_benchmark.py, tracing run_seed() and score_fixture() to understand how the static CRITIC_COVERAGE and BASELINE_COVERAGE values reach the three reports. Add tests showing that changing a captured response changes its score and that missing responses, provenance, or scorer/rubric identity cannot produce a report. Done means each scored cell has a provenance-complete manifest, while existing reports remain rejected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.