openai/monitorability-evals

Open-sourced evaluation suite from the Monitoring Monitorability paper

Stars: 98
Forks: 17
Open beginner issues: 0
Indexed issues: 9

Dominant language: Python
License: Apache-2.0
Last GitHub push: Aug 17, 2026
Latest indexed: Sep 19, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

9 open issues indexed

Intervention pairs can compare different outcome targets under the same instance ID Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

openai/monitorability-evals#55 · Opened Aug 21, 2026
Intervention bootstrap can omit reporting groups from individual replicates Open

Difficulty 3/5 1-2 days Newbie friendliness 72/100

openai/monitorability-evals#54 · Opened Aug 21, 2026
Row caps can split intervention pairs and make smoke-run metrics undefined Open

Difficulty 3/5 1-2 days Newbie friendliness 70/100

openai/monitorability-evals#53 · Opened Aug 21, 2026
Process scaffold incorrectly scores specificity despite having no sound negatives Open

Difficulty 3/5 1-2 days Newbie friendliness 74/100

openai/monitorability-evals#52 · Opened Aug 20, 2026
Output cleaning coerces boolean metric fields to integer 0/1 Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

openai/monitorability-evals#51 · Opened Aug 20, 2026
Mock intervention monitor treats correct control rows as positive targets Open

Difficulty 3/5 1-2 days Newbie friendliness 76/100

openai/monitorability-evals#50 · Opened Aug 20, 2026
Bootstrap configuration accepts zero group sampling and fails after metric computation starts Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

openai/monitorability-evals#46 · Opened Aug 20, 2026
Directional reference answers are generated with inverted mock correctness semantics Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

openai/monitorability-evals#45 · Opened Aug 20, 2026
Can't find intervention/*.jsonl Open

Difficulty 2/5 1-3 hours Newbie friendliness 45/100

openai/monitorability-evals#1 · Opened Apr 24, 2026

A short digest of beginner-friendly GitHub issues.