TIGER-AI-Lab / TIGER-AI-Lab/ClawBench

Harbor: strict harbor/terminal-bench format compliance + e2b sandbox support

Open
#331 3 comments 0 reactions 1 assignee View on GitHub

@Perry2004 is already working on this.

Since Aug 31, 2026.

Dominant language
Python
Stars
795
Forks
58
Avg merge
2d 21h
Merged PRs (30d)
18

Description

Goal

Make ClawBench a first-class Harbor benchmark, strictly following the conventions used by harbor-framework/harbor and harbor-framework/terminal-bench, including running on Harbor's remote sandboxes (-e e2b) — not just local Docker.

Today clawbench-harbor-adapt (added in v0.7.0, #232) produces Harbor tasks that load fine on Harbor 0.22, but we are not yet aligned with the ecosystem end-to-end.

Requirements

1. Strict task format (terminal-bench parity)
  • Canonical layout per task: task.toml ([task] name = "<org>/<name>", authors, keywords), instruction.md, environment/ (or [environment].docker_image), tests/test.sh → numeric /logs/verifier/reward.txt + reward.json, solution/solve.sh (documented no-op — live sites have no scripted oracle)
  • dataset.toml manifest with per-task digests (harbor add --scan / Packager content hash)
  • Every task loads with Harbor TaskConfig in CI; committed dataset regenerates deterministically (scripts/harbor/regenerate.sh, validate-harbor.yml — on branch feat/harbor-hub-registry)
  • Numeric-only reward.json metrics (depends on #324 — should merge first)
2. Registry + Hub (like terminal-bench)
  • Root registry.json so harbor run --repo TIGER-AI-Lab/ClawBench -d clawbench-v2 works
  • Publish tiger-ai-lab/clawbench-v2 to the Harbor Hub (harbor publish --public), tags per release
  • Prebuilt runtime image on GHCR (clawbench-harbor-runtime:<version>) so tasks ship no build context
3. e2b (and other remote sandbox) support
  • harbor run ... -e e2b completes a smoke task end-to-end (also verify -e daytona/modal are not broken)
  • No dependency on local podman/docker specifics: image pullable from a public registry, no bind mounts outside the task dir, healthcheck-driven readiness (Chromium + CDP + interceptor)
  • Judge + PurelyMail credentials passed via --ve/--env-file only; document resource floor (1 CPU / 2 GB per trial)
  • Prior note: docs/harbor-0.20-port-notes.md (branch feat/harbor-0.20) verified the E2B path needs no local disk — fold conclusions into docs/harbor.md
4. Upstream adapter
  • adapters/clawbench/ package in the official harbor adapter init layout (README template sections, adapter_metadata.json, parity_experiment.json, run_clawbench.yaml, src/.../{adapter,main}.py with --output-dir --limit --overwrite --task-ids)
  • Parity experiment: same agent both sides (native clawbench-batch vs harbor run), ≥3 runs/side, mean ± sample SEM, ranges overlap
  • PRs: [WIP] Adapter: clawbench → harbor-datasets datasets/clawbench/ → harbor registry.json[Ready for Review]

Starting point

Branch feat/harbor-hub-registry already implements most of §1–§2 and the adapter package skeleton (129 prebuilt-mode tasks, 7 MB, all pass Harbor 0.22 TaskConfig; digests match harbor add). §3 (e2b) and the parity run in §4 are the open work.

Review

@Perry2004 — you built the original Harbor compatibility layer in #232; please review the format decisions here (strict layout, prebuilt image, registry/Hub split) and the e2b plan.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.