Calibrate the Eliciting-Requirements Evaluation Baseline

Purpose

Complete the independent human calibration required before the Agent Eval Harness baseline for #56 is treated as trusted.

Dependency

Blocked by #56 and PR #64 landing the skill, corpus, judges, and recorded baseline.

Scope

At least two independent reviewers assess a documented sample from both development and regression partitions against property-based references, not exact wording. The sample must cover all six EARS patterns, complex-pattern splitting, false-ready rejection, both readiness contracts, consistency analysis, implementation leakage, uncertainty, host independence, and controlled-language findings.

Deliverables

• Reviewer scores and critical-failure flags per sampled case.
• Agreement analysis with deterministic and semantic judges.
• Adjudication notes identifying rubric, judge, skill, or corpus corrections.
• A committed calibration artifact with reviewer, date, model/settings, harness revision, thresholds, cost, and latency metadata.
• Confirmed failures added to the held-out regression corpus before changing the skill or promoting a new baseline.

Acceptance criteria

• The sample selection and reviewer assignments are recorded before scoring.
• Any false-ready result is a critical failure regardless of aggregate score.
• The prior baseline remains available and per-case evidence is retained.
• The baseline is not marked trusted until calibration and adjudication complete.

Start by reading issue #56 and PR #64, which are prerequisites for the skill, corpus, judges, and recorded baseline. After they land, review the development and regression partitions and record sample selection and reviewer assignments before scoring. Done means the calibration artifact, agreement and adjudication evidence, retained prior baseline, and confirmed regression failures are committed before trust is promoted.