ContextLab / ContextLab/orchestrator

Calibrate and risk-tier independent artifact review

Open
#490 0 comments 0 reactions 0 assignees View on GitHub
architecture enhancement
Dominant language
Python
Stars
3
Forks
2
Avg merge
13m
Merged PRs (30d)
1

Description

## Origin

Deferred from the MVP proposed as a fresh realization of #485.

## Goal

Evolve the MVP's bounded plan/final-output review into a calibrated, risk-tiered review service.

## Scope

- Per reviewer model/prompt/version precision, recall, false-positive rate, abstention, and cost metrics.
- Seeded-defect and adjudicated production datasets for calibration.
- Risk tiers: deterministic checks, sampling/differential review, mandatory independent review, and human adjudication.
- Model-family versus evidence-channel independence experiments.
- Dispute handling and accepted-risk policy.
- Review of intermediate artifacts where measured benefit justifies cost.

## Acceptance criteria

- [ ] Reviewers are evaluated against a versioned benchmark and production adjudications.
- [ ] Policy selects review depth from consequence/risk and measured reviewer quality.
- [ ] Review-loop non-convergence and cost stay within declared bounds.
- [ ] A pass reports residual risk; it never claims that an artifact is simply "clean."

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.