TIGER-AI-Lab / TIGER-AI-Lab/ClawBench

judge_llm.py has diverged from judge.py (no Gemini support, wrong responses endpoint, 800-token cap) — collapse the duplication

Open
#300 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
795
Forks
58
Avg merge
2d 21h
Merged PRs (30d)
18

Description

src/clawbench/runner/judge_llm.py:20-22 says it is a "drop-in replacement" for judge.py with the same interface. It has diverged in ways that matter, and it is the module behind the lenient rubric that produces the published Reward-lenient column:

judge.py judge_llm.py
judge_context kwarg (passed at run.py:587-597) yes missing (:169-171)
openai-responses api_type correct endpoint routed to /chat/completions (:180-181)
google-generative-ai (the /v1beta/openai fix) supported (:99-108, :217-220) absent
max_tokens 4096, with a comment that reasoning judges burn hidden tokens (:123) 800 (:114)

The default judge is deepseek-v4-pro, a reasoning model — an 800-token budget risks truncating its verdict, and truncation lands on the except path whose lenient default is match=True (see #295). On top of that, _post_json / _build_user_msg / _call_* are ~100 duplicated lines that now have to be fixed twice; the Gemini fix already only landed in one copy.

Ask: reduce judge_llm.py to its JUDGE_SYSTEM prompt and delegate to judge.py's _run_judge / _build_user_msg with the rubric as a parameter. That restores identical api_type coverage, token budget, and signature in one place.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/clawbench/runner/judge_llm.py and compare its wrapper with judge.py, especially _run_judge and _build_user_msg and the call sites in run.py:587-597. Reduce the duplicate implementation while preserving the rubric parameter; done means both judge paths share the same interface, API endpoint coverage, Gemini support, and 4096-token budget.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.