Validate the judge reward signal with human calibration and domain-specific evaluators

The analysis notes that the SDK’s reward model is conceptually clear but still evidence-light: the paper specifies intent, adherence, and completion judges, but does not show that those judges actually correlate with human judgment or domain success. That is a direct risk to the learning loop because reward misspecification can cause the learner to optimize the wrong behavior.

This ticket should turn the current reward design from a plausible architecture into a validated measurement layer.

Suggested scope

Add a reward validation study that measures judge-to-human agreement.
Add support for custom evaluators so teams can plug in domain-specific checks.
Document how to evaluate whether the default judge weights are appropriate for a given workload.
Persist raw per-metric values and make it easy to reweight historical episodes without re-running judges, which the whitepaper already points to as an advantage.

Acceptance criteria

A documented calibration workflow exists for the default judges.
The SDK supports pluggable evaluators without changing the learning core.
A sample report shows how judge scores correlate with human review or task-success outcomes.

Start by locating the SDK’s current reward design, default judges, and learning-core interfaces, then read the whitepaper sections describing intent, adherence, completion, and reweighting. The work is done when a documented calibration workflow, pluggable evaluators, raw per-metric episode values, and a sample human-correlation report are supported without changing the learning core.

Written by the indexing model from the issue text.