TIGER-AI-Lab / TIGER-AI-Lab/ClawBench
feat: adapter — run InternLM/WildClawBench (60 in-the-wild OpenClaw tasks) under the ClawBench harness
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 795
- Forks
- 58
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 18
Description
Goal
Let the ClawBench harness load and execute tasks from InternLM/WildClawBench (60 original in-the-wild tasks; agents run in a live OpenClaw personal-assistant environment; multi-step tool orchestration + multimodal evidence; arXiv 2605.10912; HF internlm/WildClawBench; strongest model ~62%) so users can:
- Run a long-horizon, tool-heavy agent eval through our 5-layer trace pipeline + interception schema, without standing up the OpenClaw stack separately.
- Share traces in our existing
*TraceHF dataset format so comparisons across benchmarks live in one place.
Scope
- Environment bridge — package the OpenClaw runtime as an optional container layer (
Dockerfile.openclaw-style) the harness can boot per-batch. Reuse our existing harness-isolation pattern. - Task loader —
clawbench.corpus.adapters.wildclawbenchingests theirtasks/directory + HF metadata; maps theiragencyvsmultimodalaxes onto our category enum. - Tool-call accounting — their tasks chain 10–60+ tool calls; our
actions.jsonlalready captures these but we need a counter exposed inrun-meta.jsonto match their reported metrics. - Scoring passthrough — wire to their checker code in
eval/so leaderboard-comparable scores come out of the same run. - CLI —
clawbench run --corpus wildclawbench --model <m>end-to-end.
Why now
Different axis from ours (long-horizon agency + multimodal vs. our single-request interception). Demonstrating that our harness can host both makes the case for a common evaluation infrastructure across the agent-benchmark ecosystem.
Acceptance
-
clawbench run --corpus wildclawbench --limit 5 --model <m>boots OpenClaw, runs 5 tasks, produces 5-layer trace bundles. - Scores within ±3pp of the upstream eval/ pipeline on a sampled subset.
- Docs:
eval/adapters/wildclawbench.md+ OpenClaw container README.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing harness-isolation pattern and corpus adapter structure, then inspect the WildClawBench tasks/ and eval/ code referenced in the issue. Trace how actions.jsonl and run-meta.json are produced before planning the OpenClaw container bridge. Done means the --limit 5 command runs five tasks, emits trace bundles and comparable scores, with the two named documentation files added.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, huggingface, python
- Domain
- ai, devops, documentation, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100