PolicyEngine / PolicyEngine/policybench

Explanation contract is unenforced — set the v2 scoring policy

Open
#145 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2
Forks
3
Avg merge
12h 7m
Merged PRs (30d)
13

Description

From the August 2026 harness audit (policy decision needed before the #139 v2 freeze).

Symptom

The response contract requires a short explanation per output, but the requirement is not enforced at scoring time: blank-explanation answers score normally.

Evidence (verified)

  • 61 blank-explanation rows accepted on the current board: grok-4.3 = 56, kimi-k2.6 = 4, haiku-4.5 = 1.
  • If blanks scored as misses, grok-4.3 would drop 2.574 pts (~7 ranks). No other model moves materially.

Decision for v2

Two coherent options:

  1. Score on numbers only (recommended): keep the explanation request in the prompt, publish explanation-compliance as a per-model column, but don't gate the accuracy score on prose compliance. Scoring prose mixes constructs into a numeric-accuracy headline.
  2. Enforce as miss, stated up front alongside the parse-failure policy (like Sonnet 5's 56/1,984 unparsed-as-miss under auto).

Either way the v2 methodology must state the policy explicitly; today's behavior is option 1 by accident, undocumented.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the August 2026 harness audit and issue #139's v2 freeze context. Review the current scoring behavior for blank explanations and compare it with the documented parse-failure policy. Done means the v2 methodology explicitly selects and documents one explanation-scoring policy, including how explanation compliance is reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.