zivtech / zivtech/react-critic
fix: benchmark reports score static coverage maps, not model outputs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 59m
- Merged PRs (30d)
- 2
Description
Problem
At commit 5ed06f6e2667835912dc1b6cf1c40cc7da6dd07c, scripts/run_benchmark.py does not invoke a candidate or baseline model and does not read captured model responses. run_seed() calls score_fixture() twice against the static CRITIC_COVERAGE and BASELINE_COVERAGE dictionaries, then publishes those derived values as harsh/baseline scores in the three benchmark reports.
The external-skill evidence audit therefore retains the React, Next.js, and React Native result sets as rejected-integrity. They are rubric-coverage projections, not evidence of model behavior.
Required repair
- Make benchmark scoring consume immutable captured candidate and baseline outputs, with provider/model/parameters and input/output hashes.
- Fail closed when either response, its provenance, or the scorer/rubric identity is missing.
- Keep static rubric-coverage analysis separate and label it as static analysis if it remains useful.
- Add tests proving that changing a captured response can change the score and that missing responses cannot produce a benchmark report.
- Emit a provenance-complete run manifest for each scored cell.
Acceptance boundary
Repairing the runner does not rehabilitate the existing reports. They remain rejected until a separately approved run produces complete captured outputs and manifests. This issue does not authorize paid model calls.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in scripts/run_benchmark.py, tracing run_seed() and score_fixture() to understand how the static CRITIC_COVERAGE and BASELINE_COVERAGE values reach the three reports. Add tests showing that changing a captured response changes its score and that missing responses, provenance, or scorer/rubric identity cannot produce a report. Done means each scored cell has a provenance-complete manifest, while existing reports remain rejected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100