TIGER-AI-Lab / TIGER-AI-Lab/ClawBench
[P1] reproduce: validate sample coverage and selected rubrics before reporting PASS/FAIL
@plox-sumit is already working on this.
Since Sep 17, 2026.
- Dominant language
- Python
- Stars
- 795
- Forks
- 58
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 18
Description
Problem
The reproduction verdict compares three percentages but ignores the observed/published task counts. Meanwhile --rubric lenient and --rubric strict are supported options, yet missing unrequested metrics are replaced with zero and still compared. This can accept mismatched samples and reject a correct single-rubric reproduction.
Sources: verdict(), summary-to-verdict conversion.
Verified offline examples
from clawbench.eval.reproduce import verdict, PUBLISHED_V2_HERMES
verdict((50, 40, 20, 5), (50, 40, 20, 130), 2)[0]
# True: same percentages are accepted despite 5 versus 130 tasks.
verdict((54.6, 44.6, 0, 130), PUBLISHED_V2_HERMES['claude-opus-4-7'], 2)[0]
# False: a matching lenient-only result fails against the uncomputed strict column.
The second example models the exact summary.get('reward_pct_strict', 0) behavior for --rubric lenient.
Acceptance criteria
- Verify nonzero expected task coverage before declaring a full reproduction; compare task identities/revision when available, not only count.
- Compare only explicitly requested rubric columns; display omitted metrics as not evaluated rather than 0%.
- Distinguish incomplete/invalid evidence from a measured mismatch, including missing required metrics and judge-inconclusive counts.
- Test both single-rubric modes, full mode, sample-count mismatch, missing metrics, and empty input.
Suggested priority P1: scientific result validity. This is a verdict contract bug, not the historical corpus-count issue #244 or path portability #296. The numbers above exercise checked-in reference data, not a claim about today's live leaderboard.
@Perry2004 please review the definition of a valid reproduction and its exit-status semantics.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.