TIGER-AI-Lab / TIGER-AI-Lab/ClawBench
feat: adapter — run WebVoyager (live-website, screenshot+LLM-judge browser agent) under the ClawBench harness
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 795
- Forks
- 58
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 18
Description
Goal
Let the ClawBench harness load + execute tasks from MinorJerry/WebVoyager (1.1k★) — the closest scope-peer of ClawBench: live websites, multimodal browser agent, screenshot+LLM-judge scoring. Paper: "WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models."
Hosting their tasks under our 5-layer trace pipeline lets the community directly compare two scoring paradigms on the same agent runs: their screenshot judge vs our HTTP-interception + payload judge.
Scope
- Task loader —
clawbench.corpus.adapters.webvoyageringests theirdata/WebVoyager_data.jsonltask list (URL + natural-language goal + reference answer); normalizes into our task schema. - Scoring passthrough — wire to their screenshot-judge prompt so we can emit both scores (ours: interception+payload judge; theirs: screenshot judge). Side-by-side comparison surfaces what each metric catches/misses.
- Trace bundle conformance — capture
recording.mp4,actions.jsonl,agent-messages.jsonl,requests.jsonl,interception.json,run-meta.jsonper task (already standard). - CLI —
clawbench run --corpus webvoyager --model <m>end-to-end.
Why now
WebVoyager is the most-cited live-website agent benchmark. A clean comparison ("WebVoyager screenshot-judge 59% vs ClawBench interception-judge 54.7%") would meaningfully strengthen our /compare page (#180) and the related-work narrative.
Acceptance
-
clawbench run --corpus webvoyager --limit 5 --model claude-opus-4-7produces 5-layer trace bundles per task - Both scoring outputs in
run-meta.json:webvoyager_judge_scoreand ourintercepted/judge_match - Score within ±3pp of the upstream
gpt-4-1106-preview-runs.zipreproduction on a sampled subset - Docs:
eval/adapters/webvoyager.mdwalkthrough
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the clawbench.corpus.adapters.webvoyager entry point and inspect data/WebVoyager_data.jsonl to understand the task fields and normalization requirements. Trace the existing clawbench run --corpus CLI and standard trace bundle outputs, then review the screenshot-judge integration requirements. Done means the documented command produces five-layer bundles, both scores appear in run-meta.json, sampled results meet the stated tolerance, and eval/adapters/webvoyager.md explains the workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100