TIGER-AI-Lab / TIGER-AI-Lab/ClawBench

feat: adapter — run InternLM/WildClawBench (60 in-the-wild OpenClaw tasks) under the ClawBench harness

Open
#171 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
795
Forks
58
Avg merge
2d 21h
Merged PRs (30d)
18

Description

Goal

Let the ClawBench harness load and execute tasks from InternLM/WildClawBench (60 original in-the-wild tasks; agents run in a live OpenClaw personal-assistant environment; multi-step tool orchestration + multimodal evidence; arXiv 2605.10912; HF internlm/WildClawBench; strongest model ~62%) so users can:

  1. Run a long-horizon, tool-heavy agent eval through our 5-layer trace pipeline + interception schema, without standing up the OpenClaw stack separately.
  2. Share traces in our existing *Trace HF dataset format so comparisons across benchmarks live in one place.

Scope

  • Environment bridge — package the OpenClaw runtime as an optional container layer (Dockerfile.openclaw-style) the harness can boot per-batch. Reuse our existing harness-isolation pattern.
  • Task loaderclawbench.corpus.adapters.wildclawbench ingests their tasks/ directory + HF metadata; maps their agency vs multimodal axes onto our category enum.
  • Tool-call accounting — their tasks chain 10–60+ tool calls; our actions.jsonl already captures these but we need a counter exposed in run-meta.json to match their reported metrics.
  • Scoring passthrough — wire to their checker code in eval/ so leaderboard-comparable scores come out of the same run.
  • CLIclawbench run --corpus wildclawbench --model <m> end-to-end.

Why now

Different axis from ours (long-horizon agency + multimodal vs. our single-request interception). Demonstrating that our harness can host both makes the case for a common evaluation infrastructure across the agent-benchmark ecosystem.

Acceptance

  • clawbench run --corpus wildclawbench --limit 5 --model <m> boots OpenClaw, runs 5 tasks, produces 5-layer trace bundles.
  • Scores within ±3pp of the upstream eval/ pipeline on a sampled subset.
  • Docs: eval/adapters/wildclawbench.md + OpenClaw container README.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing harness-isolation pattern and corpus adapter structure, then inspect the WildClawBench tasks/ and eval/ code referenced in the issue. Trace how actions.jsonl and run-meta.json are produced before planning the OpenClaw container bridge. Done means the --limit 5 command runs five tasks, emits trace bundles and comparable scores, with the two named documentation files added.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, huggingface, python
Domain
ai, devops, documentation, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.