PolicyEngine / PolicyEngine/chronicle

Register raw microdata releases as source artifacts: licence/access manifest fields, hash-only registration, first registrations

Open
#221 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
0
Forks
1
Avg merge
11h 48m
Merged PRs (30d)
45

Description

Implements the raw-microdata identity ADR (docs/adr-chronicle-raw-microdata-identity.md, PR opened alongside this issue): Chronicle registers every raw microdata release its consumers build from, stores bytes only for redistributable public-use files, and never parses microdata into rows or facts.

Why now

  • Microcosm's raw inputs are pinned in four different places with no shared registry, licence record, or witnessed fetch time: per-country source_stages.json (UK: 19 licensed_microdata + 2 private_microdata entries carry sha256; US: 8 of 37 public_microdata entries carry sha256, 29 do not; BE: the restricted_microdata entry carries none), the ASEC raw-stage checkpoint metadata, code constants (e.g. weeks_unemployed.py), and build_us_multispine_pool.py --*-sha256 arguments. None references a Chronicle artifact.
  • Publishers revise and withdraw microdata (IRS PUF withdrawn 2026; Census reissues under the same vintage label). A registered hash with a witnessed fetch time is the only durable statement that a release existed with those bytes.
  • The node-graph rebuild (microcosm#836) declares identity per node; the roots of the graph should resolve to one witnessed registration.

Tasks

  • Manifest schema: add licence (publisher terms id or URL) and access (public | licensed | restricted, closed set) to manifest.yaml; validate in validate-package / artifact inventory. Today 1 of 159 manifests carries a license field.
  • Hash-only registration: fetch-artifact / publish-raw (or a sibling register-artifact) must accept a licensed/restricted entry with checksum, size, vintage, access route and refuse bytes; public entries archive bytes under raw/{source_id}/{package_id}/{year}/{sha256}/{filename} as today.
  • Registration is manifest-level: confirm no harness gate demands parsed source rows/cells for a microdata package, or add the explicit carve-out.
  • First registrations, from Microcosm's manifests: US public-use files (CPS ASEC pooled vintages, CPS basic monthly 2024, ACS PUMS household/person, SCF 2022, SIPP 2023) with bytes; IRS PUF 2015, FRS 2023-24 tabs, SPI donor, BE-SILC 2023 hash-only. Exact list = every *_microdata artifact kind in packages/microcosm-build/src/microcosm/build/*/source_stages.json.
  • Docs: storage-architecture "Object Key Conventions" example for a microdata release; harness doc section on hash-only entries.
  • Consumer side: PolicyEngine/microcosm issue (linked below) adds chronicle_artifact references and sha256 to every microdata artifact entry and fails closed on mismatch.

Boundary

Unchanged in substance: no reconciliation, aging, imputation, target activation, or microdata-derived facts. The June 30 removal (#68) of microdata parsers, adapters, and tracked raw storage stands; this adds manifests and two manifest fields only. Licensed bytes never enter a Chronicle store or credential scope.

Related: #143 (ledger→chronicle surface migration; the ledger-raw bucket name is migrating under mechanism 3, registrations should target whichever bucket is current at landing time), microcosm#530 (pin raw PUF), microcosm#665 (UK raw ingest, workstream E), microcosm#204.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/adr-chronicle-raw-microdata-identity.md, then inspect manifest.yaml validation in validate-package/artifact inventory and the fetch-artifact and publish-raw entry points. Trace the source_stages.json files under packages/microcosm-build/src/microcosm/build/* and review the storage-architecture and harness documentation. Done means manifest licence/access fields validate, hash-only registrations refuse bytes, public registrations archive bytes, and the listed microdata releases are registered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.