Create a cross-project evaluation corpus for requirements and Job Site conformance

Purpose

Create a versioned, representative evaluation corpus for ProtoBot's
requirements elicitation and autonomous Job Site flow.

The corpus should make it possible to answer two separate questions:

1. Did ProtoBot convert a free-form project description into a complete,
   correct, implementable set of requirements?
2. Can the Job Site turn a known-good requirements set into a working,
   behaviorally conformant prototype?

The combined pipeline should also be runnable, but its scores must remain
attributable to the elicitation and building stages separately.

Scope

• Define a versioned fixture format containing:
  • a free-form project description;
  • golden Vision and Architecture artifacts;
  • a golden set of EARS requirements;
  • interface and runtime metadata;
  • reference behavior tests and evaluation metadata.
• Seed representative projects, initially including:
  • a command-line application;
  • a server-side API application;
  • a web application.
• Build or integrate evaluation runners for:
  • semantic comparison of generated requirements against the golden set;
  • Job Site execution from golden requirements;
  • end-to-end execution from the free-form description.
• Record reproducible inputs, outputs, model/settings metadata, traces, scores,
  failures, and corpus revisions.
• Keep holdout behavior tests outside the generator-visible fixture inputs so
  the Job Site cannot optimize directly against the evaluator.

Evaluation rules

• Compare requirements by behavior, coverage, applicability, and
  implementability, not exact wording.
• Treat missing requirements, unsupported additions, false readiness, and
  implementation leakage as explicit failure categories.
• Use the golden requirements as the stable Job Site input baseline.
• Score prototype behavior with hidden conformance tests appropriate to each
  interface type.
• Preserve per-case evidence so aggregate scores cannot hide critical failures.
• Calibrate human or semantic judges before promoting a corpus revision as a
  trusted baseline.

Dependencies

• Blocked by #63 for the calibrated requirements-evaluation baseline and
  judging conventions.
• Blocked by #69 for the deterministic Worker/Triage fixture execution path.
• Blocked by #79 for the executable Job Site sandbox contract.
• Related to #66 and #77, which provide the local vertical slice and
  replayable Drafting Table flow that this corpus should exercise.

Out of scope

• Production applications or production-quality code.
• Benchmarking model latency or cost as the primary objective.
• Supporting every language, framework, or interface type in the first corpus
  revision.
• Replacing the existing requirements-skill corpus in #63.
• Making exact generated source-code comparison the success criterion.

Acceptance criteria

• The fixture schema and versioning policy are documented.
• At least three projects covering CLI, server-side API, and web application
  interfaces are committed with free-form descriptions and reviewed golden
  artifacts.
• Each project has behavior-level reference tests, including hidden holdout
  tests for Job Site evaluation.
• A headless run produces separate requirements, Job Site, and end-to-end
  results with per-case evidence.
• Repeated evaluation of unchanged inputs is reproducible within documented
  nondeterminism limits.
• A corpus change records rationale, reviewer/calibration evidence, and its
  effect on the prior baseline.
• The corpus can run locally without a hosted cluster or external service
  credentials.

Start by reading the dependency issues #63, #69, and #79, then review related work in #66 and #77 to understand the available evaluation and execution paths. Define the versioned fixture schema, seed CLI, server-side API, and web projects, and specify separate reproducible requirements, Job Site, and end-to-end results with per-case evidence. Done means the documented corpus runs locally with hidden holdout tests and records revision rationale and baseline impact.