TIGER-AI-Lab / TIGER-AI-Lab/ClawBench

[P1] reproduce: validate sample coverage and selected rubrics before reporting PASS/FAIL

Open
#362 1 comment 0 reactions 1 assignee View on GitHub

@plox-sumit is already working on this.

Since Sep 17, 2026.

bug
Dominant language
Python
Stars
795
Forks
58
Avg merge
2d 21h
Merged PRs (30d)
18

Description

Problem

The reproduction verdict compares three percentages but ignores the observed/published task counts. Meanwhile --rubric lenient and --rubric strict are supported options, yet missing unrequested metrics are replaced with zero and still compared. This can accept mismatched samples and reject a correct single-rubric reproduction.

Sources: verdict(), summary-to-verdict conversion.

Verified offline examples

from clawbench.eval.reproduce import verdict, PUBLISHED_V2_HERMES
verdict((50, 40, 20, 5), (50, 40, 20, 130), 2)[0]
# True: same percentages are accepted despite 5 versus 130 tasks.
verdict((54.6, 44.6, 0, 130), PUBLISHED_V2_HERMES['claude-opus-4-7'], 2)[0]
# False: a matching lenient-only result fails against the uncomputed strict column.

The second example models the exact summary.get('reward_pct_strict', 0) behavior for --rubric lenient.

Acceptance criteria

  • Verify nonzero expected task coverage before declaring a full reproduction; compare task identities/revision when available, not only count.
  • Compare only explicitly requested rubric columns; display omitted metrics as not evaluated rather than 0%.
  • Distinguish incomplete/invalid evidence from a measured mismatch, including missing required metrics and judge-inconclusive counts.
  • Test both single-rubric modes, full mode, sample-count mismatch, missing metrics, and empty input.

Suggested priority P1: scientific result validity. This is a verdict contract bug, not the historical corpus-count issue #244 or path portability #296. The numbers above exercise checked-in reference data, not a claim about today's live leaderboard.

@Perry2004 please review the definition of a valid reproduction and its exit-status semantics.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.