PolicyEngine / PolicyEngine/microcosm

Column closure: every artifact column claimed by exactly one lineage class (spec totality)

Open
#697 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Max, 2026-08-15: the process diagrams should have revealed the geography flaw sooner — the observability has to be total, not just cover the parts we extracted.

The measurement

Column-closure audit of the f025 artifact against the lineage spec (#695), 2026-08-15:

  • 392 columns across 6 entities; spec claims 156 (+10 structural) ≈ 42%; 226 unclaimed.
  • In the unclaimed dark matter: ~187 raw source (caps) columns; the 3 geography columns (where the #696 flaw hid); ~12 takes_up_* flags — stochastic draws with policy consequences, declared nowhere; capital-gains-tail donor receipts; spine-source ids and role/filing inputs.

Inventory: one JSON fixture of entity/column names extracted from the artifact (names only, no data).

The increment

  1. Total lineage classes in specs/us_imputation_lineage.yaml: measured_native (per source channel), imputed (existing), computed (existing), assigned_by_ladder (#696 lane adds), take_up_draw (new — each flag's distribution, conditioning, parameter source, and calibration status), receipt, structural.
  2. Column-closure conformance test: the artifact column inventory is a committed fixture; every column must be claimed by exactly one class entry; unclaimed or doubly-claimed ⇒ test failure. Inventory regenerated by a tool from any built artifact; drift between fixture and a fresh build fails loudly.
  3. Coverage meter on /lineage: claimed/total rendered in the hero, per class — the number that keeps us honest.
  4. Immediate deep-dive owed: the take_up_* mechanism (where drawn, from what) documented into the spec as part of this increment — it is currently exactly as invisible as geography was.

Sequencing: lands after the two wave-1 lanes (#696 block-first, per-family predictor sets) merge, since all three edit the same spec file. Owner runs it as its own PR.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with specs/us_imputation_lineage.yaml and the committed artifact column inventory, then locate the tool that regenerates the inventory from a built artifact and the /lineage rendering. Run the column-closure conformance test first. Done means every artifact column has exactly one lineage class, fixture/build drift fails, take_up_* behavior is documented, and coverage is shown per class.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, yaml
Domain
data-engineering, observability, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.