TIGER-AI-Lab / TIGER-AI-Lab/ClawBench

feat: adapter — run WebVoyager (live-website, screenshot+LLM-judge browser agent) under the ClawBench harness

Open
#190 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
795
Forks
58
Avg merge
2d 21h
Merged PRs (30d)
18

Description

Goal

Let the ClawBench harness load + execute tasks from MinorJerry/WebVoyager (1.1k★) — the closest scope-peer of ClawBench: live websites, multimodal browser agent, screenshot+LLM-judge scoring. Paper: "WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models."

Hosting their tasks under our 5-layer trace pipeline lets the community directly compare two scoring paradigms on the same agent runs: their screenshot judge vs our HTTP-interception + payload judge.

Scope

  • Task loaderclawbench.corpus.adapters.webvoyager ingests their data/WebVoyager_data.jsonl task list (URL + natural-language goal + reference answer); normalizes into our task schema.
  • Scoring passthrough — wire to their screenshot-judge prompt so we can emit both scores (ours: interception+payload judge; theirs: screenshot judge). Side-by-side comparison surfaces what each metric catches/misses.
  • Trace bundle conformance — capture recording.mp4, actions.jsonl, agent-messages.jsonl, requests.jsonl, interception.json, run-meta.json per task (already standard).
  • CLIclawbench run --corpus webvoyager --model <m> end-to-end.

Why now

WebVoyager is the most-cited live-website agent benchmark. A clean comparison ("WebVoyager screenshot-judge 59% vs ClawBench interception-judge 54.7%") would meaningfully strengthen our /compare page (#180) and the related-work narrative.

Acceptance

  • clawbench run --corpus webvoyager --limit 5 --model claude-opus-4-7 produces 5-layer trace bundles per task
  • Both scoring outputs in run-meta.json: webvoyager_judge_score and our intercepted/judge_match
  • Score within ±3pp of the upstream gpt-4-1106-preview-runs.zip reproduction on a sampled subset
  • Docs: eval/adapters/webvoyager.md walkthrough

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the clawbench.corpus.adapters.webvoyager entry point and inspect data/WebVoyager_data.jsonl to understand the task fields and normalization requirements. Trace the existing clawbench run --corpus CLI and standard trace bundle outputs, then review the screenshot-judge integration requirements. Done means the documented command produces five-layer bundles, both scores appear in run-meta.json, sampled results meet the stated tolerance, and eval/adapters/webvoyager.md explains the workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.