Automattic / Automattic/wp-gym
Collect large-N calibration distributions across model tiers
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
Parent: #236
## Problem
The pilot now has real live rows, cheap-model rows, repeated-attempt rows, and calibration metadata, but it does not yet have enough large-N evidence to establish stable task difficulty or score reliability.
Dan's review standard calls for calibrated distributions, not just proof that the loop can emit valid rows.
## Scope
- Run no-op, scripted/reference, cheap-model, frontier-model, repeated-attempt, and human/reference rows where applicable.
- Record pass-rate bands, confidence intervals, variance, cost, timing, and failure classes by task/model tier.
- Separate calibration rows from headline benchmark rows.
- Keep benchmark-mode gates fail-closed until distributions are statistically meaningful.
## Acceptance criteria
- At least one candidate task family has large-N distributions across model tiers.
- Scenario metadata points to stable result sets with confidence intervals and variance.
- Registry reports show row counts, pass@1/pass@n, reward distributions, and failure classes by model tier.
- Docs explain when a pilot task can graduate from calibration evidence to benchmark-ready scoring.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with parent issue #236 and inspect the existing pilot, scenario metadata, registry reports, and benchmark-mode gates. Determine how calibration rows and benchmark rows are currently represented before planning large-N runs across the listed model tiers. Done means one task family has stable distributions, reports include the requested statistics and failure classes, and the documentation defines graduation criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- data, machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100