ContextLab / ContextLab/orchestrator
Calibrate and risk-tier independent artifact review
- Dominant language
- Python
- Stars
- 3
- Forks
- 2
- Avg merge
- 13m
- Merged PRs (30d)
- 1
Description
## Origin
Deferred from the MVP proposed as a fresh realization of #485.
## Goal
Evolve the MVP's bounded plan/final-output review into a calibrated, risk-tiered review service.
## Scope
- Per reviewer model/prompt/version precision, recall, false-positive rate, abstention, and cost metrics.
- Seeded-defect and adjudicated production datasets for calibration.
- Risk tiers: deterministic checks, sampling/differential review, mandatory independent review, and human adjudication.
- Model-family versus evidence-channel independence experiments.
- Dispute handling and accepted-risk policy.
- Review of intermediate artifacts where measured benefit justifies cost.
## Acceptance criteria
- [ ] Reviewers are evaluated against a versioned benchmark and production adjudications.
- [ ] Policy selects review depth from consequence/risk and measured reviewer quality.
- [ ] Review-loop non-convergence and cost stay within declared bounds.
- [ ] A pass reports residual risk; it never claims that an artifact is simply "clean."
Contributor guide
Assessment
This issue has not been assessed yet.