TIGER-AI-Lab / TIGER-AI-Lab/ClawBench

[P1] rescore: stale judge caches are silently attributed to a newly requested judge model

Open
#363 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
795
Forks
58
Avg merge
2d 21h
Merged PRs (30d)
18

Description

Problem

Both pending selection and rescore_one reuse any cached non-null verdict, without comparing judge identity/configuration, rubric revision, instruction or intercepted evidence. The aggregate then unconditionally writes roll['judge_model'] = args.judge_model. Requesting a new judge can therefore produce a report labeled with that judge even when every verdict came from an older judge.

Sources: cache reuse, pending selection, aggregate attribution.

Verified reproduction

Create one intercepted run with judge.json = {"match": true, "judge_model": "old-judge"}; run rescore.main() with --judge-model new-judge --rubric strict, a temporary dummy model configuration, and a mocked judge function.

Observed: exit 0, judge calls 0, rescore-summary.json.judge_model = "new-judge", n_match_strict = 1. Direct rescore_one(..., judge_model='new-judge', force=False) returns the old-judge object unchanged. No network call was made.

Acceptance criteria

  • Define a non-secret evaluation fingerprint covering judge identity/config, rubric/prompt revision, instruction and intercepted input.
  • Reuse only a valid, matching cache; handle legacy cache entries explicitly (recompute or report unknown provenance).
  • Preserve actual per-verdict provenance and reject or explicitly describe mixed-provenance aggregates.
  • Keep --force as an explicit recomputation option; ordinary model changes must not silently mislabel results.
  • Test changing model, input and rubric revision, and unchanged-cache reuse.

Suggested priority P1: misleading benchmark attribution. Related to provenance PR #347, but run metadata alone does not fix this evaluator cache contract. #313 fixes paths/config loading; #319 fixes judge transports; neither addresses cache identity.

@Perry2004 please review the cache/provenance contract and reporting requirements.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/clawbench/eval/rescore.py at the cache reuse, pending selection, and aggregate paths linked in the issue; trace rescore_one and rescore.main with the reproduced old-judge/new-judge case. Add focused coverage for changed model, intercepted input, rubric revision, and unchanged-cache reuse, then verify rescore-summary.json preserves actual provenance and handles legacy or mixed results explicitly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.