PolicyEngine / PolicyEngine/policybench
Explanation contract is unenforced — set the v2 scoring policy
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 3
- Avg merge
- 12h 7m
- Merged PRs (30d)
- 13
Description
From the August 2026 harness audit (policy decision needed before the #139 v2 freeze).
Symptom
The response contract requires a short explanation per output, but the requirement is not enforced at scoring time: blank-explanation answers score normally.
Evidence (verified)
- 61 blank-explanation rows accepted on the current board: grok-4.3 = 56, kimi-k2.6 = 4, haiku-4.5 = 1.
- If blanks scored as misses, grok-4.3 would drop 2.574 pts (~7 ranks). No other model moves materially.
Decision for v2
Two coherent options:
- Score on numbers only (recommended): keep the explanation request in the prompt, publish explanation-compliance as a per-model column, but don't gate the accuracy score on prose compliance. Scoring prose mixes constructs into a numeric-accuracy headline.
- Enforce as miss, stated up front alongside the parse-failure policy (like Sonnet 5's 56/1,984 unparsed-as-miss under auto).
Either way the v2 methodology must state the policy explicitly; today's behavior is option 1 by accident, undocumented.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the August 2026 harness audit and issue #139's v2 freeze context. Review the current scoring behavior for blank explanations and compare it with the documented parse-failure policy. Done means the v2 methodology explicitly selects and documents one explanation-scoring policy, including how explanation compliance is reported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100