PolicyEngine / PolicyEngine/chronicle
Register raw microdata releases as source artifacts: licence/access manifest fields, hash-only registration, first registrations
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- Avg merge
- 11h 48m
- Merged PRs (30d)
- 45
Description
Implements the raw-microdata identity ADR (docs/adr-chronicle-raw-microdata-identity.md, PR opened alongside this issue): Chronicle registers every raw microdata release its consumers build from, stores bytes only for redistributable public-use files, and never parses microdata into rows or facts.
Why now
- Microcosm's raw inputs are pinned in four different places with no shared registry, licence record, or witnessed fetch time: per-country
source_stages.json(UK: 19licensed_microdata+ 2private_microdataentries carry sha256; US: 8 of 37public_microdataentries carry sha256, 29 do not; BE: therestricted_microdataentry carries none), the ASEC raw-stage checkpoint metadata, code constants (e.g.weeks_unemployed.py), andbuild_us_multispine_pool.py --*-sha256arguments. None references a Chronicle artifact. - Publishers revise and withdraw microdata (IRS PUF withdrawn 2026; Census reissues under the same vintage label). A registered hash with a witnessed fetch time is the only durable statement that a release existed with those bytes.
- The node-graph rebuild (microcosm#836) declares identity per node; the roots of the graph should resolve to one witnessed registration.
Tasks
- Manifest schema: add
licence(publisher terms id or URL) andaccess(public|licensed|restricted, closed set) tomanifest.yaml; validate invalidate-package/ artifact inventory. Today 1 of 159 manifests carries alicensefield. - Hash-only registration:
fetch-artifact/publish-raw(or a siblingregister-artifact) must accept alicensed/restrictedentry with checksum, size, vintage, access route and refuse bytes;publicentries archive bytes underraw/{source_id}/{package_id}/{year}/{sha256}/{filename}as today. - Registration is manifest-level: confirm no harness gate demands parsed source rows/cells for a microdata package, or add the explicit carve-out.
- First registrations, from Microcosm's manifests: US public-use files (CPS ASEC pooled vintages, CPS basic monthly 2024, ACS PUMS household/person, SCF 2022, SIPP 2023) with bytes; IRS PUF 2015, FRS 2023-24 tabs, SPI donor, BE-SILC 2023 hash-only. Exact list = every
*_microdataartifact kind inpackages/microcosm-build/src/microcosm/build/*/source_stages.json. - Docs: storage-architecture "Object Key Conventions" example for a microdata release; harness doc section on hash-only entries.
- Consumer side: PolicyEngine/microcosm issue (linked below) adds
chronicle_artifactreferences and sha256 to every microdata artifact entry and fails closed on mismatch.
Boundary
Unchanged in substance: no reconciliation, aging, imputation, target activation, or microdata-derived facts. The June 30 removal (#68) of microdata parsers, adapters, and tracked raw storage stands; this adds manifests and two manifest fields only. Licensed bytes never enter a Chronicle store or credential scope.
Related: #143 (ledger→chronicle surface migration; the ledger-raw bucket name is migrating under mechanism 3, registrations should target whichever bucket is current at landing time), microcosm#530 (pin raw PUF), microcosm#665 (UK raw ingest, workstream E), microcosm#204.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/adr-chronicle-raw-microdata-identity.md, then inspect manifest.yaml validation in validate-package/artifact inventory and the fetch-artifact and publish-raw entry points. Trace the source_stages.json files under packages/microcosm-build/src/microcosm/build/* and review the storage-architecture and harness documentation. Done means manifest licence/access fields validate, hash-only registrations refuse bytes, public registrations archive bytes, and the listed microdata releases are registered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100