TIGER-AI-Lab / TIGER-AI-Lab/ClawBench

feat: adapter — run evolvent-ai/ClawMark (100 multi-day coworker-agent tasks, rule-based scoring) under the ClawBench harness

Open
#169 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
795
Forks
58
Avg merge
2d 21h
Merged PRs (30d)
18

Description

Goal

Let the ClawBench harness load and execute tasks from evolvent-ai/ClawMark (100 multi-day, multimodal, multi-environment coworker-agent tasks across 13 professional domains; each task simulates 1–3 working days; strict rule-based Python checkers, zero LLM-as-judge → fully reproducible) so users can:

  1. Run a long-horizon, time-staged eval through our 5-layer trace pipeline.
  2. Get a corpus with deterministic scoring as a baseline alongside our judge-based two-stage rubric.

Scope

  • Stage scheduler — tasks unfold across 1–3 stages (Mon 3/16 → Tue 3/17 → Wed 3/18). Add a stage-advance hook to the harness so the per-day instruction file is loaded and the agent only sees the current day's context.
  • Backend mocks — package their backends (filesystem, GreenMail, Notion mock, Google Sheets mock, Radicale CalDAV) as a docker-compose layer the harness can spin up per batch.
  • Implicit-state injection — their environment mutates between stages (new email arrives, DB rows update, calendar shifts). Hook this into the existing per-task extra_info/ mount lifecycle so state changes fire at the right boundary.
  • Multimodal fixturesassets/input/ ships screenshots, PDFs, audio, video. Just mount it; existing harness already supports binary fixtures.
  • Deterministic scorer — invoke their 10–25 Python checker functions per task and emit a normalized run-meta.json score field. Skip our LLM judge for this corpus (their explicit design choice).
  • CLIclawbench run --corpus clawmark --model <m> end-to-end.

Why now

Their leaderboard already covers the frontier (Sonnet 4.6 at 0.758, Opus 4.6 at 0.746, GPT-5.4 at 0.720). Hosting that corpus under our harness gives our users a deterministic, reproducible alternative metric — and demonstrates that our trace pipeline plays well with rule-based scorers, not just LLM-judge pipelines.

Acceptance

  • clawbench run --corpus clawmark --limit 5 --model claude-sonnet-4-6 produces 5-layer trace bundles with stage boundaries marked in actions.jsonl.
  • Deterministic checker scores match their upstream scripts/ output exactly on a sampled subset.
  • Docs: eval/adapters/clawmark.md + multi-day stage diagram.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing per-task extra_info/ mount lifecycle, binary fixture support, and the clawbench run CLI entry point. Compare ClawMark's scripts/ checker output and stage data with the harness trace format, including run-meta.json and actions.jsonl. Done means the stated clawbench command runs five tasks, preserves stage boundaries, matches sampled checker scores, and adds eval/adapters/clawmark.md with the multi-day diagram.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, python
Domain
ai, cli, devops, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.