TIGER-AI-Lab / TIGER-AI-Lab/ClawBench
feat: adapter — run evolvent-ai/ClawMark (100 multi-day coworker-agent tasks, rule-based scoring) under the ClawBench harness
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 795
- Forks
- 58
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 18
Description
Goal
Let the ClawBench harness load and execute tasks from evolvent-ai/ClawMark (100 multi-day, multimodal, multi-environment coworker-agent tasks across 13 professional domains; each task simulates 1–3 working days; strict rule-based Python checkers, zero LLM-as-judge → fully reproducible) so users can:
- Run a long-horizon, time-staged eval through our 5-layer trace pipeline.
- Get a corpus with deterministic scoring as a baseline alongside our judge-based two-stage rubric.
Scope
- Stage scheduler — tasks unfold across 1–3 stages (Mon 3/16 → Tue 3/17 → Wed 3/18). Add a stage-advance hook to the harness so the per-day instruction file is loaded and the agent only sees the current day's context.
- Backend mocks — package their backends (filesystem, GreenMail, Notion mock, Google Sheets mock, Radicale CalDAV) as a docker-compose layer the harness can spin up per batch.
- Implicit-state injection — their environment mutates between stages (new email arrives, DB rows update, calendar shifts). Hook this into the existing per-task
extra_info/mount lifecycle so state changes fire at the right boundary. - Multimodal fixtures —
assets/input/ships screenshots, PDFs, audio, video. Just mount it; existing harness already supports binary fixtures. - Deterministic scorer — invoke their 10–25 Python checker functions per task and emit a normalized
run-meta.jsonscore field. Skip our LLM judge for this corpus (their explicit design choice). - CLI —
clawbench run --corpus clawmark --model <m>end-to-end.
Why now
Their leaderboard already covers the frontier (Sonnet 4.6 at 0.758, Opus 4.6 at 0.746, GPT-5.4 at 0.720). Hosting that corpus under our harness gives our users a deterministic, reproducible alternative metric — and demonstrates that our trace pipeline plays well with rule-based scorers, not just LLM-judge pipelines.
Acceptance
-
clawbench run --corpus clawmark --limit 5 --model claude-sonnet-4-6produces 5-layer trace bundles with stage boundaries marked inactions.jsonl. - Deterministic checker scores match their upstream
scripts/output exactly on a sampled subset. - Docs:
eval/adapters/clawmark.md+ multi-day stage diagram.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing per-task extra_info/ mount lifecycle, binary fixture support, and the clawbench run CLI entry point. Compare ClawMark's scripts/ checker output and stage data with the harness trace format, including run-meta.json and actions.jsonl. Done means the stated clawbench command runs five tasks, preserves stage boundaries, matches sampled checker scores, and adds eval/adapters/clawmark.md with the multi-day diagram.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, python
- Domain
- ai, cli, devops, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100