TIGER-AI-Lab / TIGER-AI-Lab/ClawBench
judge_llm.py has diverged from judge.py (no Gemini support, wrong responses endpoint, 800-token cap) — collapse the duplication
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 795
- Forks
- 58
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 18
Description
src/clawbench/runner/judge_llm.py:20-22 says it is a "drop-in replacement" for judge.py with the same interface. It has diverged in ways that matter, and it is the module behind the lenient rubric that produces the published Reward-lenient column:
judge.py |
judge_llm.py |
|
|---|---|---|
judge_context kwarg (passed at run.py:587-597) |
yes | missing (:169-171) |
openai-responses api_type |
correct endpoint | routed to /chat/completions (:180-181) |
google-generative-ai (the /v1beta/openai fix) |
supported (:99-108, :217-220) |
absent |
max_tokens |
4096, with a comment that reasoning judges burn hidden tokens (:123) |
800 (:114) |
The default judge is deepseek-v4-pro, a reasoning model — an 800-token budget risks truncating its verdict, and truncation lands on the except path whose lenient default is match=True (see #295). On top of that, _post_json / _build_user_msg / _call_* are ~100 duplicated lines that now have to be fixed twice; the Gemini fix already only landed in one copy.
Ask: reduce judge_llm.py to its JUDGE_SYSTEM prompt and delegate to judge.py's _run_judge / _build_user_msg with the rubric as a parameter. That restores identical api_type coverage, token budget, and signature in one place.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/clawbench/runner/judge_llm.py and compare its wrapper with judge.py, especially _run_judge and _build_user_msg and the call sites in run.py:587-597. Reduce the duplicate implementation while preserving the rubric parameter; done means both judge paths share the same interface, API endpoint coverage, Gemini support, and 4096-token budget.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100