TIGER-AI-Lab / TIGER-AI-Lab/ClawBench
Harbor: strict harbor/terminal-bench format compliance + e2b sandbox support
@Perry2004 is already working on this.
Since Aug 31, 2026.
- Dominant language
- Python
- Stars
- 795
- Forks
- 58
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 18
Description
Goal
Make ClawBench a first-class Harbor benchmark, strictly following the conventions used by harbor-framework/harbor and harbor-framework/terminal-bench, including running on Harbor's remote sandboxes (-e e2b) — not just local Docker.
Today clawbench-harbor-adapt (added in v0.7.0, #232) produces Harbor tasks that load fine on Harbor 0.22, but we are not yet aligned with the ecosystem end-to-end.
Requirements
1. Strict task format (terminal-bench parity)
- Canonical layout per task:
task.toml([task] name = "<org>/<name>", authors, keywords),instruction.md,environment/(or[environment].docker_image),tests/test.sh→ numeric/logs/verifier/reward.txt+reward.json,solution/solve.sh(documented no-op — live sites have no scripted oracle) -
dataset.tomlmanifest with per-task digests (harbor add --scan/ Packager content hash) - Every task loads with Harbor
TaskConfigin CI; committed dataset regenerates deterministically (scripts/harbor/regenerate.sh,validate-harbor.yml— on branchfeat/harbor-hub-registry) - Numeric-only
reward.jsonmetrics (depends on #324 — should merge first)
2. Registry + Hub (like terminal-bench)
- Root
registry.jsonsoharbor run --repo TIGER-AI-Lab/ClawBench -d clawbench-v2works - Publish
tiger-ai-lab/clawbench-v2to the Harbor Hub (harbor publish --public), tags per release - Prebuilt runtime image on GHCR (
clawbench-harbor-runtime:<version>) so tasks ship no build context
3. e2b (and other remote sandbox) support
-
harbor run ... -e e2bcompletes a smoke task end-to-end (also verify-e daytona/modalare not broken) - No dependency on local podman/docker specifics: image pullable from a public registry, no bind mounts outside the task dir, healthcheck-driven readiness (Chromium + CDP + interceptor)
- Judge + PurelyMail credentials passed via
--ve/--env-fileonly; document resource floor (1 CPU / 2 GB per trial) - Prior note:
docs/harbor-0.20-port-notes.md(branchfeat/harbor-0.20) verified the E2B path needs no local disk — fold conclusions intodocs/harbor.md
4. Upstream adapter
-
adapters/clawbench/package in the officialharbor adapter initlayout (README template sections,adapter_metadata.json,parity_experiment.json,run_clawbench.yaml,src/.../{adapter,main}.pywith--output-dir --limit --overwrite --task-ids) - Parity experiment: same agent both sides (native
clawbench-batchvsharbor run), ≥3 runs/side, mean ± sample SEM, ranges overlap - PRs:
[WIP] Adapter: clawbench→ harbor-datasetsdatasets/clawbench/→ harborregistry.json→[Ready for Review]
Starting point
Branch feat/harbor-hub-registry already implements most of §1–§2 and the adapter package skeleton (129 prebuilt-mode tasks, 7 MB, all pass Harbor 0.22 TaskConfig; digests match harbor add). §3 (e2b) and the parity run in §4 are the open work.
Review
@Perry2004 — you built the original Harbor compatibility layer in #232; please review the format decisions here (strict layout, prebuilt image, registry/Hub split) and the e2b plan.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.