PolicyEngine / PolicyEngine/microcosm

Root identity: every raw microdata input hash-pinned and referenced to its Chronicle registration

Open
#848 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement tier:standard
Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Consumer side of the Chronicle raw-microdata identity ADR (PolicyEngine/chronicle docs/adr-chronicle-raw-microdata-identity.md, PR and implementation issue opened today in that repo): every raw microdata input Microcosm builds from should resolve to one witnessed Chronicle registration, by hash, and the build should fail closed when local bytes differ.

Current state (verified 2026-09-02 on origin/main d2b31496)

Manifest Kind Entries With sha256 Chronicle reference
us/source_stages.json public_microdata 37 8 none
us/source_stages.json versioned_derived_microdata 2 1 none
uk/source_stages.json licensed_microdata 19 19 none
uk/source_stages.json, uk/hmrc_income_source_stages.json private_microdata 2 2 none
be/source_stages.json restricted_microdata 1 0 none

Pins that do exist live in four places: these manifests, the ASEC raw-stage checkpoint metadata (us_runtime/asec_checkpoint.py validates per-source sha256/member_sha256), code constants (us_runtime/weeks_unemployed.py), and tools/build_us_multispine_pool.py --*-sha256 arguments. No shared registry, no licence record, no witnessed fetch time.

Proposal

  1. Every *_microdata artifact entry in a country source_stages.json carries sha256 and a chronicle_artifact reference (source_id, package_id, year, sha256, and the R2 key when bytes exist). The raw-only doctrine (microcosm#530, #665 scope ruling) already requires pinning the raw licensed asset; this makes the pin a shared, witnessed one.
  2. source_manifest.py validates the new fields; the source runtime records the resolved registrations in the build manifest / Logbook next to the Chronicle consumer-artifact pin it already records.
  3. A fail-closed gate: local bytes (or the checkpoint's recorded per-source hashes) must equal the registered sha256; mismatch stops the build with the publisher, vintage, and both hashes in the message.
  4. Licensed and restricted files (FRS, SPI, BE-SILC, IRS PUF) reference hash-only registrations. Bytes stay where they are today (licensed environments, private HF repos); nothing about data access changes.
  5. New countries land registered from day one (Armenia #814 is the first test).

Acceptance

  • Zero *_microdata entries without sha256 + chronicle_artifact across all country manifests; a contract test enforces it.
  • Build manifests list every root registration; the gate has a test for the mismatch path.
  • No on-disk or artifact key renames (frozen at v1 per #639); no change to HF destinations.

Related: #530, #665 (workstream E), #204, #814, #836 (node identity), chronicle#143 (bucket name migrating; reference whichever bucket is current).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with source_manifest.py and the country source_stages.json manifests, then read the ASEC checkpoint validation in us_runtime/asec_checkpoint.py and the build_us_multispine_pool.py hash arguments. Trace how the source runtime records build-manifest and Logbook data. Done means every *_microdata entry has the required pin and Chronicle reference, contract coverage exists, and the hash-mismatch gate test passes without renaming artifacts or changing HF destinations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.