PolicyEngine / PolicyEngine/chronicle

Flag cross-source inconsistencies as a first-class diagnostic (report, never reconcile)

Open
#75 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
1
Avg merge
11h 48m
Merged PRs (30d)
45

Description

What

Add cross-source inconsistency flagging as a first-class Ledger diagnostic — reported, never reconciled.

Ledger rightly refuses to reconcile sources. Flagging that two source-backed claims disagree is not reconciling, and Ledger is the only layer where both sources coexist with enough structure (semantic fact keys, universe constraints, periods, assertions) to check. Consumers then decide.

Motivating failures this would have caught

  • The May-19 eCPS calibration break (policyengine-us-data#1107 / the downstream-dashboard incident): in the $500k+ AGI bracket, the calibrated amount tripled while the count fell — a count-versus-amount consistency violation visible in the target data itself.
  • SOI TY2022 itemized levels versus TPC's FY2026 projection (~$99B vs ~$130B on the RCC dashboard): a fact-versus-source-projection gap that is expected (aging) but should be measured and reported, not discovered downstream.

Shape

A ledger check-consistency pass over a bundle or consumer artifact that emits a report (never mutates facts):

  1. Count-versus-amount coherence within a series: for bracketed families with both count and amount measures, flag period-over-period moves where amount and count diverge beyond a threshold in opposite directions.
  2. Same-semantic-fact disagreement: rows sharing a semantic_fact_key (or same concept/entity/geography/period universe) from different sources whose values disagree beyond a tolerance — e.g. SOI vs CBO income tax receipts for the same period.
  3. Observation-versus-source-projection gaps: for series with both (e.g. CBO projected vs Treasury actual receipts), report realized gaps — useful to Thesis directly as forecaster-evaluation data.

Output: a consistency_report.json in bundle/artifact builds with per-finding severity, the fact keys involved, and the comparison arithmetic — same spirit as coverage.json. No reconciliation, no imputation, no edits.

Refs: the facts-only ADR (docs/adr-ledger-facts-only.md), #71, policyengine-us-data#1107.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/adr-ledger-facts-only.md and issue #71, then review the existing bundle or consumer artifact checks and the coverage.json report shape. Done means a ledger check-consistency pass emits consistency_report.json with severity, involved fact keys, and comparison arithmetic without mutating, reconciling, or imputing facts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.