Support running other tasks suits under ClawBench's infrastructure using adaptor layers.

Proposal

Add a task-import adapter so ClawBench can run tasks defined by other browser-agent benchmarks — specifically claw-eval and wild-claw-bench to start, with a clear registration path for additional benchmarks — without forking their task repos.

Why

• Every browser-agent benchmark defines tasks in slightly different schemas (WebArena-style JSON, Mind2Web step traces, VisualWebArena starting-state dumps, WorkArena goal-conditions, etc.). Today a research team that wants to reuse our submission-interception + five-layer recording has to hand-convert each task.
• Both claw-eval and wild-claw-bench target the same problem space we do (live-site agentic tasks). Running them through our harness gives a common scoring substrate — one judge pipeline, one recording format, one leaderboard row.
• This is cheap: our task schema (task.json with url, instruction, time_limit, rubric pointers) is a near-superset of what most benchmarks encode. The adapter is import-only; no outbound export needed yet.

Current state

• Our tasks live under test-cases/<category>/<name>/task.json, loaded by src/clawbench/run.py via paths.bundled_data_dir().
• clawbench cases lists bundled cases only. There is no mechanism to point at an external task dir or a third-party benchmark repo.
• No schema-conversion layer exists; task.json is parsed directly.

Proposed design

1. Adapter layout

src/clawbench/adapters/
  __init__.py               # registry: {"clawbench-native": load_native, "claw-eval": load_claw_eval, ...}
  native.py                 # identity loader for our own task.json
  claw_eval.py              # claw-eval → ClawBenchTask
  wild_claw_bench.py        # wild-claw-bench → ClawBenchTask
  webarena.py               # web-arena-x/webarena + visualwebarena → ClawBenchTask
  mind2web.py               # OSU-NLP-Group/Mind2Web + Online-Mind2Web → ClawBenchTask
  webvoyager.py             # MinorJerry/WebVoyager → ClawBenchTask
  workarena.py              # ServiceNow/WorkArena goals → ClawBenchTask
  browsergym.py             # ServiceNow/BrowserGym generic envs → ClawBenchTask
  webcanvas.py              # iMeanAI/WebCanvas → ClawBenchTask
  agentbench.py             # THUDM/AgentBench web-shopping subset → ClawBenchTask
  theagentcompany.py        # TheAgentCompany (CMU/reka) work tasks → ClawBenchTask
  schema.py                 # Pydantic ClawBenchTask dataclass (shared target type)
  _base.py                  # AdapterBase + AdapterWarning + registry decorator

Each adapter exports load(path: Path) -> list[ClawBenchTask]. Fields that don't have a 1:1 mapping (e.g. stepwise rubrics vs our end-state rubric) get a clear AdapterWarning on load with a link to the field-mapping table in the module docstring.

2. CLI surface

clawbench run <case>                                # existing: bundled native task
clawbench run --source claw-eval --task <id>        # import single external task
clawbench run --source webarena --task shopping.42
clawbench batch --source wild-claw-bench:/path/to/repo --cases 1-50
clawbench batch --source mind2web --cases all
clawbench cases --source claw-eval                  # list importable tasks from a source
clawbench sources                                   # list registered adapters + status
clawbench sources refresh <name>                    # git pull the cached checkout

--source accepts either a registered name (resolves to a default checkout location under $XDG_CACHE_HOME/clawbench/sources/<name>) or <name>:<path> for an explicit clone.

3. Source fetch

For registered sources, lazy-clone on first use via git clone --depth 1 into the cache dir. Respect CLAWBENCH_OFFLINE=1 (no network) — error clearly if cache is empty in offline mode. Pin each adapter to a specific upstream commit SHA (stored in adapters/_pins.yaml) so an upstream rename doesn't silently break our runs.

4. Scoring contract

Adapters MUST declare which scoring layers apply:

• submission_intercept — needs a final write request; not every external benchmark has one.
• end_state_dom_match — our default judge pipeline.
• step_trace_replay — for Mind2Web-style benchmarks where the rubric is per-step.
• goal_predicate — for WorkArena/BrowserGym style boolean goal functions.
• llm_judge_only — when the upstream rubric is a free-form prompt to a judge model (WebVoyager).

Unsupported layers emit score=null in the recording rather than a zero, so leaderboard aggregation doesn't conflate "failed" with "not scored."

5. Field-mapping contract

Shared ClawBenchTask (Pydantic) target fields and how each external source maps in:

Field	claw-eval	wild-claw-bench	webarena	mind2web	webvoyager	workarena
url	start_url	url	start_url	first step URL	web	env init URL
instruction	task	nl_instruction	intent	confirmed_task	ques	goal NL gen
time_limit	direct	direct	default 300s	default 300s	default 300s	env-derived
category	direct	tag	sites[0]	domain	web_name	instance class
rubric	direct	direct	eval_types	action_reprs	LLM-judge prompt	evaluate() fn
submission_intercept	yes if has_write=true	yes if is_write_task	inferable from eval_types	no (replay)	no	no

Each adapter writes this table into its module docstring. When upstream drops a field, loader emits AdapterWarning with the upstream commit SHA, the missing field name, and a one-line default.

Scope

In scope (this issue)

• Read-only import of claw-eval, wild-claw-bench, webarena, mind2web, webvoyager — the five with clearest schema overlap.
• Shared ClawBenchTask Pydantic type.
• AdapterWarning emission per field-mapping gap with upstream SHA + field name.
• clawbench sources CLI subcommand.
• Adapter unit tests with fixtures (10-20 sample tasks per source).

Follow-ups (separate issues)

• workarena, browsergym, webcanvas, agentbench, theagentcompany adapters.
• Export path (ClawBench → external format) — inverse direction, only after import path ships.
• Upstream schema-drift CI job that runs adapter loaders against pinned SHAs weekly.
• Per-adapter leaderboard sub-rows on the public leaderboard.

Out of scope

• Writing our tasks back to external formats.
• Tight coupling with upstream schema changes — adapters pin to a specific upstream commit/tag.
• Rubric translation when the semantic contract differs (e.g. Mind2Web's per-step oracle ≠ our end-state judge); we run what we can and emit score=null for the rest.

Open questions

• License compatibility — verify each upstream benchmark's license permits in-our-container execution.
  • claw-eval, wild-claw-bench: TBD (confirm before shipping)
  • WebArena: Apache 2.0 — OK
  • Mind2Web: CC-BY 4.0 — OK
  • WebVoyager: MIT — OK
  • WorkArena: Apache 2.0 — OK
  • BrowserGym: Apache 2.0 — OK
• Provenance in the trace — propose top-level source key in the run manifest and per-task source_id so leaderboard can partition by origin.
• Leaderboard gating — do external-source runs count toward the headline ClawBench score, or live on a separate tab? Recommend separate tab with source: badge; headline stays pure-native for comparability.
• Instance-agnostic vs instance-specific tasks — WorkArena/BrowserGym tasks often need a live instance (e.g. a ServiceNow demo instance). Do we bundle a docker-compose to spin one up, or skip those tasks? Recommend skip with AdapterWarning("requires external instance").
• Starting-state fixtures — some benchmarks (VisualWebArena) ship initial DOM / cookie / localStorage snapshots. Our runner loads from a URL; extending to accept a fixture bundle is a bigger refactor than this issue covers. Defer with AdapterWarning("starting-state fixtures not supported").

Implementation plan (incremental, one PR per bullet)

1. Land src/clawbench/adapters/_base.py + schema.py + native.py + __init__.py + clawbench sources CLI command. No behavior change for bundled tasks.
2. Land claw_eval.py + wild_claw_bench.py with unit fixtures.
3. Land webarena.py + mind2web.py with fixtures.
4. Land webvoyager.py + fixtures.
5. Land upstream-SHA pinning (_pins.yaml) + a GitHub Actions job that validates pinned SHAs weekly.
6. Follow-ups (separate PRs, separate issues): remaining adapters, export path.

Priority

Medium-high. This is a research-ecosystem multiplier — enables external teams to reuse our recording/judge infra without inventing a parallel one — and it's a concrete answer to the "yet another benchmark" critique that any new benchmark faces. Not on the critical path to our own leaderboard, but high-leverage once the five named adapters land.

Related

• Supersedes any narrower "just support claw-eval" scoping.
• Cross-references: #70 (hermes-agent harness — independent work but shares the 3-file plugin pattern), #71 (token-cost display — orthogonal), #67 (symlink packaging — adapters add src/clawbench/adapters/ which is a real directory, no symlink concerns), #69 (TUI cross-platform tests — each new adapter gets a smoke test entry).

Start by reading src/clawbench/run.py, paths.bundled_data_dir(), and the existing task.json loading path. The incremental plan begins with src/clawbench/adapters/_base.py, schema.py, native.py, init.py, and the clawbench sources command, followed by five adapters and fixture tests. Done for the first step means registered native loading and source listing work without changing bundled-task behavior.