TIGER-AI-Lab / TIGER-AI-Lab/ClawBench
[P1] rescore: stale judge caches are silently attributed to a newly requested judge model
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 795
- Forks
- 58
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 18
Description
Problem
Both pending selection and rescore_one reuse any cached non-null verdict, without comparing judge identity/configuration, rubric revision, instruction or intercepted evidence. The aggregate then unconditionally writes roll['judge_model'] = args.judge_model. Requesting a new judge can therefore produce a report labeled with that judge even when every verdict came from an older judge.
Sources: cache reuse, pending selection, aggregate attribution.
Verified reproduction
Create one intercepted run with judge.json = {"match": true, "judge_model": "old-judge"}; run rescore.main() with --judge-model new-judge --rubric strict, a temporary dummy model configuration, and a mocked judge function.
Observed: exit 0, judge calls 0, rescore-summary.json.judge_model = "new-judge", n_match_strict = 1. Direct rescore_one(..., judge_model='new-judge', force=False) returns the old-judge object unchanged. No network call was made.
Acceptance criteria
- Define a non-secret evaluation fingerprint covering judge identity/config, rubric/prompt revision, instruction and intercepted input.
- Reuse only a valid, matching cache; handle legacy cache entries explicitly (recompute or report unknown provenance).
- Preserve actual per-verdict provenance and reject or explicitly describe mixed-provenance aggregates.
- Keep
--forceas an explicit recomputation option; ordinary model changes must not silently mislabel results. - Test changing model, input and rubric revision, and unchanged-cache reuse.
Suggested priority P1: misleading benchmark attribution. Related to provenance PR #347, but run metadata alone does not fix this evaluator cache contract. #313 fixes paths/config loading; #319 fixes judge transports; neither addresses cache identity.
@Perry2004 please review the cache/provenance contract and reporting requirements.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/clawbench/eval/rescore.py at the cache reuse, pending selection, and aggregate paths linked in the issue; trace rescore_one and rescore.main with the reproduced old-judge/new-judge case. Add focused coverage for changed model, intercepted input, rubric revision, and unchanged-cache reuse, then verify rescore-summary.json preserves actual provenance and handles legacy or mixed results explicitly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100