PolicyEngine / PolicyEngine/microcosm
Root identity: every raw microdata input hash-pinned and referenced to its Chronicle registration
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Consumer side of the Chronicle raw-microdata identity ADR (PolicyEngine/chronicle docs/adr-chronicle-raw-microdata-identity.md, PR and implementation issue opened today in that repo): every raw microdata input Microcosm builds from should resolve to one witnessed Chronicle registration, by hash, and the build should fail closed when local bytes differ.
Current state (verified 2026-09-02 on origin/main d2b31496)
| Manifest | Kind | Entries | With sha256 |
Chronicle reference |
|---|---|---|---|---|
us/source_stages.json |
public_microdata |
37 | 8 | none |
us/source_stages.json |
versioned_derived_microdata |
2 | 1 | none |
uk/source_stages.json |
licensed_microdata |
19 | 19 | none |
uk/source_stages.json, uk/hmrc_income_source_stages.json |
private_microdata |
2 | 2 | none |
be/source_stages.json |
restricted_microdata |
1 | 0 | none |
Pins that do exist live in four places: these manifests, the ASEC raw-stage checkpoint metadata (us_runtime/asec_checkpoint.py validates per-source sha256/member_sha256), code constants (us_runtime/weeks_unemployed.py), and tools/build_us_multispine_pool.py --*-sha256 arguments. No shared registry, no licence record, no witnessed fetch time.
Proposal
- Every
*_microdataartifact entry in a countrysource_stages.jsoncarriessha256and achronicle_artifactreference (source_id,package_id,year,sha256, and the R2 key when bytes exist). The raw-only doctrine (microcosm#530, #665 scope ruling) already requires pinning the raw licensed asset; this makes the pin a shared, witnessed one. source_manifest.pyvalidates the new fields; the source runtime records the resolved registrations in the build manifest / Logbook next to the Chronicle consumer-artifact pin it already records.- A fail-closed gate: local bytes (or the checkpoint's recorded per-source hashes) must equal the registered sha256; mismatch stops the build with the publisher, vintage, and both hashes in the message.
- Licensed and restricted files (FRS, SPI, BE-SILC, IRS PUF) reference hash-only registrations. Bytes stay where they are today (licensed environments, private HF repos); nothing about data access changes.
- New countries land registered from day one (Armenia #814 is the first test).
Acceptance
- Zero
*_microdataentries withoutsha256+chronicle_artifactacross all country manifests; a contract test enforces it. - Build manifests list every root registration; the gate has a test for the mismatch path.
- No on-disk or artifact key renames (frozen at v1 per #639); no change to HF destinations.
Related: #530, #665 (workstream E), #204, #814, #836 (node identity), chronicle#143 (bucket name migrating; reference whichever bucket is current).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with source_manifest.py and the country source_stages.json manifests, then read the ASEC checkpoint validation in us_runtime/asec_checkpoint.py and the build_us_multispine_pool.py hash arguments. Trace how the source runtime records build-manifest and Logbook data. Done means every *_microdata entry has the required pin and Chronicle reference, contract coverage exists, and the hash-mismatch gate test passes without renaming artifacts or changing HF destinations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100