PolicyEngine / PolicyEngine/policyengine.py

ensure_datasets: consume Microcosm-published UK releases (manifest-verified), so analyses can swap bases by URI

Open
#495 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7
Forks
9
Avg merge
15h 51m
Merged PRs (30d)
9

Description

Doctrine says policyengine.py is the permanent interface and Microcosm is where UK data construction is migrating (PolicyEngine/microcosm#665). The consumption path is currently missing: pe.uk.ensure_datasets materialises per-year files from hf://policyengine/policyengine-uk-data/..., and there is no way to point it at a Microcosm-published UK release (certified single-year artifact + release manifest with pinned SHA-256s).

Concrete need, with a live consumer waiting: PolicyEngine/uk-cgt-paper#6 and PolicyEngine/uk-equalising-cgt#1 — both reform analyses should rerun on the Microcosm UK build once it certifies (currently gated on microcosm#630), ideally changing nothing but a dataset URI.

Scope sketch:

  • accept a Microcosm release identifier (HF repo + release id) in ensure_datasets;
  • verify artifact SHA-256 against the release manifest before use (Microcosm releases pin these; the interface should check, matching Microcosm's own runtime verification discipline);
  • resolve the release's compatible_model_packages / compatible_core_packages constraints and fail loudly on mismatch with the installed bundle;
  • forward-year materialisation (uprating 2023 → simulation years) with the same semantics the uk-data path applies today, so results are comparable across the swap.

Related: #491 (publisher claim). Happy to pair on the design — the UK CGT arc (microcosm#552) is the first artifact that wants this path.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the pe.uk.ensure_datasets entry point and trace the existing hf://policyengine/policyengine-uk-data materialisation and forward-year uprating path. Compare it with the Microcosm release manifest and its compatible_model_packages and compatible_core_packages constraints; done means a release URI works with SHA-256 and bundle checks while preserving current semantics for the UK CGT consumer analyses.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.