Automattic / Automattic/wp-gym

Collect large-N calibration distributions across model tiers

Open
#237 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
4
PR merge metrics
No merged PRs in 30d

Description

Parent: #236

## Problem
The pilot now has real live rows, cheap-model rows, repeated-attempt rows, and calibration metadata, but it does not yet have enough large-N evidence to establish stable task difficulty or score reliability.

Dan's review standard calls for calibrated distributions, not just proof that the loop can emit valid rows.

## Scope
- Run no-op, scripted/reference, cheap-model, frontier-model, repeated-attempt, and human/reference rows where applicable.
- Record pass-rate bands, confidence intervals, variance, cost, timing, and failure classes by task/model tier.
- Separate calibration rows from headline benchmark rows.
- Keep benchmark-mode gates fail-closed until distributions are statistically meaningful.

## Acceptance criteria
- At least one candidate task family has large-N distributions across model tiers.
- Scenario metadata points to stable result sets with confidence intervals and variance.
- Registry reports show row counts, pass@1/pass@n, reward distributions, and failure classes by model tier.
- Docs explain when a pilot task can graduate from calibration evidence to benchmark-ready scoring.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with parent issue #236 and inspect the existing pilot, scenario metadata, registry reports, and benchmark-mode gates. Determine how calibration rows and benchmark rows are currently represented before planning large-N runs across the listed model tiers. Done means one task family has stable distributions, reports include the requested statistics and failure classes, and the documentation defines graduation criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
data, machine-learning, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.