TIGER-AI-Lab / TIGER-AI-Lab/ClawBench

Harbor upstream: agree live-site oracle evidence and parity acceptance protocol

Open
#359 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
795
Forks
58
Avg merge
2d 21h
Merged PRs (30d)
18

Description

Decision needed for upstream acceptance

Parent #331 proposes a documented no-op oracle because ClawBench uses live websites. Harbor's current adapter contribution guide requires solvability/oracle validation and parity with the same agents, models, and settings; it recommends discussing difficult tasks case by case. Our solve_script() only prints a message and exits successfully; it provides no intercepted request or successful judged outcome. A shell exit of 0 is not oracle evidence.

Source: https://github.com/harbor-framework/harbor/blob/fd0049126a060f930372f9b0b0aa330caf1f19e5/docs/content/docs/datasets/adapters.mdx (oracle validation, parity planning, structured reporting).

Acceptance criteria

  • Obtain and record an upstream-agreed approach for live-site solvability evidence (real reference trajectories or an explicit exception); do not label the no-op script an oracle pass.
  • Decide whether upstream adapter submission uses the conventional root layout or accepts our supported steps/run layout. Harbor's loader accepts steps; contribution acceptance is a separate question.
  • Specify a representative task subset, exact corpus/agent/model/judge revisions, browser/tool configuration, timeouts, retries, run count, and rate limits before experiments.
  • Report Stage 1, lenient/strict Stage 2, judge-inconclusive and infrastructure counts separately, with denominators and mean ± sample SEM across independent runs.
  • Produce actual adapter_metadata.json, parity_experiment.json, reference YAML, and reproducible evidence before claiming official integration.
  • Keep upstream docs-only PR harbor-framework/harbor#2495 distinct from an adapter or registered dataset submission.

This issue narrows #331 §4 rather than reopening the overall integration request. Honor Perry2004's decision to defer image publication and CI; neither is made a prerequisite here. No parity results have been fabricated or experiments launched by this issue.

@Perry2004 please review the live-site oracle approach and parity protocol and coordinate the upstream acceptance decision.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/content/docs/datasets/adapters.mdx and the current solve_script() behavior described in the issue. Resolve the upstream live-site oracle and layout decisions, then define the requested corpus, agent, model, judge, browser, and run parameters. Done means producing adapter_metadata.json, parity_experiment.json, reference YAML, and reproducible evidence without claiming integration prematurely.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, testing-qa, web-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.