PolicyEngine / PolicyEngine/microcosm

Release-delta report: explain every certified-release output change in consumer terms

Open
#366 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Problem (verbatim consumer requirement)

A high-stakes consumer (analyst producing numbers for a government office) cannot adopt populace while release-to-release output changes are unexplained in THEIR terms: "if outputs are changing frequently and each change is clearly fixing something, the only correct response is to downweight model confidence." Producer-side provenance (release manifests, supersession docs, certification comments) documents WHY inputs changed but never answers the consumer question: which of MY numbers moved, by how much, attributable to what.

Proposal: a release-delta report shipped with every certified release

For each certified release N vs N-1, an artifact (JSON + rendered md) computing, under the SAME engine version:

  • Output deltas per consumer family: the reform-validation payload rows, headline aggregates (income tax, benefit expenditures/caseloads), poverty rates, decile means — old value, new value, % change.
  • Attribution: each material delta mapped to its cause class — new/changed calibration targets (with ledger refs), method change (issue/PR refs), source-vintage refresh, selection change — the way #299's Build H comment attributes per-column changes, but automated and consumer-facing.
  • Stability summary: share of tracked outputs moving <0.5% / <2% / >5%, so "how volatile is this stack" becomes a measured, improving number rather than an impression.
  • Published in the release dir + surfaced in calibration-diagnostics.

The promotion suite (policyengine.py#462) gates fitness at flip time; this reports CHANGE between flips. Together they are the adoption contract for high-stakes consumers: certified when promoted, explained when changed.

Refs: policyengine.py#462, #299 (Build H attribution style), #305 (validation portfolio), calibration-diagnostics.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the promotion suite reference in policyengine.py#462, the Build H attribution style in #299, the validation portfolio in #305, and the calibration-diagnostics entry point. Define the release-to-release artifact around the listed consumer outputs, attribution classes, stability thresholds, and release-directory publication, with JSON and rendered Markdown produced for each certified release.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
analytics, data, release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.