TIGER-AI-Lab / TIGER-AI-Lab/ClawBench
rescore: public reproducibility CLI defaults to a maintainer's home paths and silently no-ops elsewhere
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 795
- Forks
- 58
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 18
Description
src/clawbench/eval/rescore.py:203-217 ships a maintainer's personal machine layout as the public defaults:
p.add_argument("--sweep-root", type=Path, default=Path.home() / "work/ClawBench/claw-output/sweep")
p.add_argument("--models-yaml", type=Path, default=Path.home() / "work/ClawBench/models/models.yaml")
clawbench-rescore is an installed console script (pyproject.toml:29) and both docs/scoring.md and eval/scoring.md advertise it as the way to reproduce every leaderboard number from public traces. On any machine that isn't that one, --sweep-root doesn't exist, the rglob matches nothing, and the tool prints "discovered 0 tasks" and exits 0 — a silent no-op for the exact audience we most want to succeed.
Every other code path resolves paths through WORKSPACE_ROOT / MODELS_YAML (run_support/config.py:91); rescore bypasses that.
Ask:
- Default
--models-yamlto the workspace-resolvedconfig.MODELS_YAML. - Default
--sweep-rootto./test-output(where runs actually land) or require it explicitly. - Error loudly when the root does not exist, instead of reporting 0 tasks and exiting 0.
- While there: rescore only accepts the
api_keyscalar form, rejecting theapi_keyslist form thatload_model_config(config.py:192-196) normalizes — reuseload_model_configinstead of re-parsing.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/clawbench/eval/rescore.py:203-217 and the clawbench-rescore entry in pyproject.toml:29, then compare path handling with run_support/config.py:91 and load_model_config at config.py:192-196. Verify the CLI uses workspace configuration, handles the test-output sweep root, errors when the root is missing, and accepts normalized api_keys configurations instead of silently discovering zero tasks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100