PolicyEngine / PolicyEngine/chronicle

Publish hash-pinned per-package consumer artifacts for public sources so downstream fixtures reference a Chronicle artifact, not copied rows

Open
#230 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
1
Avg merge
11h 48m
Merged PRs (30d)
45

Description

Why

Microcosm's hermetic tests carry publisher facts as hand-copied JSONL excerpts of the UK consumer feed (packages/microcosm-build/tests/fixtures/uk_target_reference_feed_rows.jsonl, documented as "public rows copied from consumer_facts_uk.jsonl"). The PolicyEngine/microcosm#834 port needs one more such excerpt: the 126 rows of hmrc-tax-free-childcare-march-2026 (9 annual-unique children with used accounts, 9 government top-up, 108 monthly used-account counts), to prove in CI that Microcosm's tax_free_childcare_spend_routed_share series equals sum(monthly children with used accounts) / annual-unique children / 12 recomputed from Chronicle's own facts. #219 rightly refused to store that share in Chronicle; the derivation is Microcosm's, the facts are Chronicle's, and the lockstep between the two should be checked against Chronicle's bytes rather than a copy someone typed.

Under the doctrine #221 states for raw microdata (consumers reference a witnessed Chronicle artifact by hash), the excerpt should be a Chronicle artifact by identity, not a copy.

Request

Publish hash-pinned per-package consumer artifacts for public source packages:

  • a build-consumer-artifact --package <package_id> (or build-suite) output written to the consumer bucket at a stable key, e.g. consumer/<source_id>/<package_id>/<artifact_sha256>/{manifest.json,consumer_facts.jsonl};
  • the manifest naming package_id, the Chronicle source commit, the artifact sha256 and the fact row count, so a downstream can vendor the two files under tests/fixtures/chronicle/<package_id>/ and assert the sha;
  • public packages only; licensed or restricted sources stay hash-only per #221.

build_source_suite(package_id, out, year=...) already builds per-package suites (it is what tests/test_childcare_source_packages.py exercises), so the gap is a published, stable location plus the package-naming manifest fields, not a new build path.

First consumers

  • PolicyEngine/microcosm#834: hmrc-tax-free-childcare-march-2026 (the routed-share derivation lockstep) and dfe-funded-early-education-childcare-2026 (the five childcare target rows).
  • PolicyEngine/microcosm#789: dft-bus05i-revenue-support-2025 once the #202 re-land is on main (the England / London / England-outside-London reconciliation fixture).

Until the artifact exists, Microcosm's lockstep test reads the same rows from its pinned UK feed when present and skips otherwise, so the check is never green by fiat.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with build_source_suite(package_id, out, year=...) and tests/test_childcare_source_packages.py to understand the existing per-package output. Trace how consumer artifacts are published to the consumer bucket, then verify that public packages produce the stable hash-based files and manifest fields for package_id, source commit, artifact sha256, and row count while restricted sources remain hash-only.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.