TIGER-AI-Lab / TIGER-AI-Lab/ClawBench

rescore: public reproducibility CLI defaults to a maintainer's home paths and silently no-ops elsewhere

Open
#296 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
795
Forks
58
Avg merge
2d 21h
Merged PRs (30d)
18

Description

src/clawbench/eval/rescore.py:203-217 ships a maintainer's personal machine layout as the public defaults:

p.add_argument("--sweep-root",   type=Path, default=Path.home() / "work/ClawBench/claw-output/sweep")
p.add_argument("--models-yaml",  type=Path, default=Path.home() / "work/ClawBench/models/models.yaml")

clawbench-rescore is an installed console script (pyproject.toml:29) and both docs/scoring.md and eval/scoring.md advertise it as the way to reproduce every leaderboard number from public traces. On any machine that isn't that one, --sweep-root doesn't exist, the rglob matches nothing, and the tool prints "discovered 0 tasks" and exits 0 — a silent no-op for the exact audience we most want to succeed.

Every other code path resolves paths through WORKSPACE_ROOT / MODELS_YAML (run_support/config.py:91); rescore bypasses that.

Ask:

  1. Default --models-yaml to the workspace-resolved config.MODELS_YAML.
  2. Default --sweep-root to ./test-output (where runs actually land) or require it explicitly.
  3. Error loudly when the root does not exist, instead of reporting 0 tasks and exiting 0.
  4. While there: rescore only accepts the api_key scalar form, rejecting the api_keys list form that load_model_config (config.py:192-196) normalizes — reuse load_model_config instead of re-parsing.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/clawbench/eval/rescore.py:203-217 and the clawbench-rescore entry in pyproject.toml:29, then compare path handling with run_support/config.py:91 and load_model_config at config.py:192-196. Verify the CLI uses workspace configuration, handles the test-output sweep root, errors when the root is missing, and accepts normalized api_keys configurations instead of silently discovering zero tasks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.