TIGER-AI-Lab / TIGER-AI-Lab/ClawBench

[P2] rescore: truncated judge cache is permanently skipped instead of retried

Open
#364 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
795
Forks
58
Avg merge
2d 21h
Merged PRs (30d)
18

Description

Problem

The CLI's pending-selection predicate parses an existing judge file inside a broad try/except Exception: continue. A truncated JSON cache therefore skips the entire run before rescore_one gets a chance to recover it. The worker already handles invalid cached JSON, but is never scheduled in this case. Verdict files are written directly, so interruption can create this state.

Sources: pending selection, worker cache fallback and direct write.

Verified reproduction

  • Create a temporary intercepted run with valid run-meta.json and data/interception.json.
  • Write the incomplete text {"match": to judge.json.
  • Run the real rescore.main() with --only-batch, --rubric strict, a temporary dummy model configuration and a mocked judge.
  • Actual: exit 0, 0 judge calls, summary reports 1 judge error. A normal rerun repeats this behavior unless the user knows to use --force or delete the cache.

Acceptance criteria

  • Treat invalid JSON, non-object verdicts and invalid verdict types as unusable caches and schedule reevaluation with a diagnostic.
  • Share cache validation between the planner and worker so their recovery behavior cannot disagree.
  • Write verdicts atomically so interruption cannot replace a valid cache with partial JSON.
  • Test CLI-level scheduling, including both rubrics where one cache is corrupt and the other is valid.
  • Define/report partial-processing errors rather than quietly treating processing as wholly successful.

Suggested priority P2: recoverability and unattended batch reliability. #299 handled match: null; this is malformed JSON rejected before that recovery path. #303 concerned batch metadata, not judge cache writes. #313 does not change this predicate.

@Perry2004 please review recovery behavior and the appropriate partial-failure exit status.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/clawbench/eval/rescore.py at the pending-selection logic around line 279, then compare it with rescore_one and the direct cache write around line 74. Reproduce the truncated judge.json case through rescore.main, then use the CLI-level scheduling scenarios in the acceptance criteria to verify invalid caches are retried, valid caches are preserved, writes are atomic, and partial failures are reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.