TIGER-AI-Lab / TIGER-AI-Lab/ClawBench
Harbor upstream: agree live-site oracle evidence and parity acceptance protocol
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 795
- Forks
- 58
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 18
Description
Decision needed for upstream acceptance
Parent #331 proposes a documented no-op oracle because ClawBench uses live websites. Harbor's current adapter contribution guide requires solvability/oracle validation and parity with the same agents, models, and settings; it recommends discussing difficult tasks case by case. Our solve_script() only prints a message and exits successfully; it provides no intercepted request or successful judged outcome. A shell exit of 0 is not oracle evidence.
Source: https://github.com/harbor-framework/harbor/blob/fd0049126a060f930372f9b0b0aa330caf1f19e5/docs/content/docs/datasets/adapters.mdx (oracle validation, parity planning, structured reporting).
Acceptance criteria
- Obtain and record an upstream-agreed approach for live-site solvability evidence (real reference trajectories or an explicit exception); do not label the no-op script an oracle pass.
- Decide whether upstream adapter submission uses the conventional root layout or accepts our supported
steps/runlayout. Harbor's loader accepts steps; contribution acceptance is a separate question. - Specify a representative task subset, exact corpus/agent/model/judge revisions, browser/tool configuration, timeouts, retries, run count, and rate limits before experiments.
- Report Stage 1, lenient/strict Stage 2, judge-inconclusive and infrastructure counts separately, with denominators and mean ± sample SEM across independent runs.
- Produce actual
adapter_metadata.json,parity_experiment.json, reference YAML, and reproducible evidence before claiming official integration. - Keep upstream docs-only PR harbor-framework/harbor#2495 distinct from an adapter or registered dataset submission.
This issue narrows #331 §4 rather than reopening the overall integration request. Honor Perry2004's decision to defer image publication and CI; neither is made a prerequisite here. No parity results have been fabricated or experiments launched by this issue.
@Perry2004 please review the live-site oracle approach and parity protocol and coordinate the upstream acceptance decision.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/content/docs/datasets/adapters.mdx and the current solve_script() behavior described in the issue. Resolve the upstream live-site oracle and layout decisions, then define the requested corpus, agent, model, judge, browser, and run parameters. Done means producing adapter_metadata.json, parity_experiment.json, reference YAML, and reproducible evidence without claiming integration prematurely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing-qa, web-dev
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100