PolicyEngine / PolicyEngine/chronicle
NZ source packages, wave 1: the populace-nz v1 16-margin surface
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- Avg merge
- 11h 48m
- Merged PRs (30d)
- 45
Description
Lane: ledger-source-ingestor (.github/chronicle-agents.yml) — judges ledger-source-fidelity + ledger-boundary
Depends on: #175 — the R2 country layout blocks every credentialed publish; the tax-year note blocks the IRD batch; the geography rulings block the MSD and MBIE batches. The stats_nz batches (PRs 1–2) can be authored first.
Context
New Zealand has no incumbent target registry to port (unlike the UK waves) — the surface being retrieved is the populace-nz v1 calibration set: 16 margins, all public, declared in docs/nz-take-up-engine-plan.md @ de7451b, with per-source URLs, licences, and fetch-status in the companion docs/nz-calibration-targets.md (every entry live-verified 2026-07-06). All values are published official statistics (Stats NZ / IRD / MSD / MBIE) — no licensed microdata is involved.
Chronicle has zero NZ presence today. This issue authors the wave-1 packages; the nz_national profile issue is drafted and will be filed separately; wave 2 (take-up/validation surface) is #177; local geography is #178.
Sourcing note (NZ-specific): stats.govt.nz's SPA, treasury.govt.nz HTML, and catalogue.data.govt.nz bot-wall scripted clients, but the file CDNs are open (ird.govt.nz, msd.govt.nz, tenancy.govt.nz, Stats NZ /assets/Uploads/…) — pin those. Infoshare and the Aotearoa Data Explorer SPA are aggregator clues, not artifacts: pin the publisher release file, or a curated CSV extract with pinned cell geometry where only the SPA publishes the cut (Statbel precedent). Manual-download registration with the canonical URL recorded is the gov.wales-precedented fallback. Infoshare retires into ADE April 2027 — nothing here may pin an Infoshare URL.
Package inventory (margin → package)
Fifteen packages serve the 16 margins (the minimum-wage half of the earnings margin is not a chronicle fact — a statutory rate has no entity population and no honest aggregation; it is a rules parameter already encoded in rulespec-nz. Stated here so the margin accounting closes: 16 margins = 15 package families + 1 rules-side parameter).
| # | Package | Artifact | Parser | Period | Geography | Margin(s) served |
|---|---|---|---|---|---|---|
| 1 | stats_nz/subnational_population_estimates_2025 |
S7 information-release xlsx (open /assets/Uploads/ CDN; direct-xlsx pattern confirmed on the Jun-2024 vintage) |
xlsx_used_range + sheets: |
calendar_year 2025, coverage 2025-06-30 |
region, REGC codes, vintage regc_2025 |
Age(5-yr)×sex×region ERP |
| 2 | stats_nz/national_population_estimates_2025 |
S8 national estimates release xlsx | xlsx_used_range |
calendar_year 2025 |
country |
Single-year-age national ERP |
| 3 | stats_nz/census_2023_households_by_region |
S2 ArcGIS hub CSV export, regional-council layer | delimited_text_full_rows |
calendar_year 2023, coverage 2023-03-07 |
region, regc_2023 |
Household composition and tenure×region (two record-set families) |
| 4 | stats_nz/census_2023_family_type |
S3 hub CSV (families/extended-families layer) | delimited_text_full_rows |
calendar_year 2023 |
country |
Family type |
| 5 | stats_nz/census_2023_ethnicity_age_region |
O5 curated CSV extract from ADE (cross-tab not published as a hub layer) | delimited_text_full_rows |
calendar_year 2023 |
region |
Ethnicity×age×region |
| 6 | ird/taxable_income_distribution_2025 |
I1 taxable-income-distribution-of-individuals-2025.xlsx |
xlsx_used_range + sheets: |
tax_year 2024 (ending-year label — #175) |
country |
Taxable income by band (count + $); age×income band (Tab 4) |
| 7 | ird/wage_salary_distribution_2025 |
I2 dataset xlsx (resolve direct URL at pin time) | xlsx_used_range |
tax_year 2024 |
country |
Wage/salary distribution |
| 8 | ird/working_for_families_statistics_sept_2025 |
I3 working-for-families-statistics---sept-2025.xlsx |
xlsx_used_range + sheets: |
tax_year 2024 |
country, entity family |
WfF families + expenditure by credit type; recipient-family income distribution (the joint-income tripwire record set) |
| 9 | ird/student_loan_statistics_march_2026 |
I4 student-loan-statistics.xlsx |
xlsx_used_range |
month 2026-03 |
country |
Student loan borrowers |
| 10 | msd/benefit_fact_sheets_national_march_2026 |
M1 national benefit tables xlsx | xlsx_used_range + sheets: |
month 2026-03 (quarter-end month; publisher quarter label in source_period_label — DWP precedent) |
country |
Main-benefit recipients by benefit × age |
| 11 | msd/benefit_fact_sheets_supplementary_march_2026 |
M1d W&I supplementary tables xlsx | xlsx_used_range |
month 2026-03 |
statistical_scope, slug ids, vintage msd_wi_region (#175 ruling) + national total row |
AS recipients by W&I region |
| 12 | msd/nzs_vp_fact_sheet_march_2026 |
O4 NZS/VP fact-sheet xlsx | xlsx_used_range |
month 2026-03 |
country |
NZS/VP recipients |
| 13 | msd/annual_report_benefit_expenses_2025 |
M2 Annual Report 2025 PDF, "Benefits or Related Expenses" schedule | pdf_text_numbers |
fiscal_year 2024 (FY2024/25 → opening year) |
country, entity government, value_scale: 1000 |
Benefit expenditure $ by program |
| 14 | mbie/tenancy_bond_rents_tla_2026 |
H3 detailed-monthly-tla-tenancy-v2.csv (open CDN; CC BY 3.0 NZ, note in package header) |
delimited_text_full_rows with selected_rows pinning one month (PIPR lane) |
month (latest stable month) |
local_authority, TA codes, vintage ta_2025 (#175 ruling) |
Rent distribution by TA × bedrooms (median; geometric mean per #175's aggregation ruling) |
| 15 | stats_nz/qes_average_earnings_march_2026 |
S12 QES Mar-2026 release tables (open CDN, not Infoshare) | xlsx_used_range |
month 2026-03 |
country; provenance_class: survey_aggregate + survey_instrument: Quarterly Employment Survey |
Average-earnings anchor (minimum wage → rules-side, see above) |
All other packages are provenance_class: administrative except #1–#5 (census) and #15 (survey_aggregate). Units: count, nzd (free-form — no vocabulary change).
Mechanics (all precedented — follow, don't reinvent)
- Per package, the 6-file pattern:
SOURCE_PACKAGE_ALIASESline (alias↔directory drift is test-enforced), artifact bytes underdb/data/…,manifest.yaml,source_package.yaml, checklist row indocs/pe-nz-source-checklist.md(may be a follow-up commit —docs/**is outside this lane), re-measuredtests/test_chronicle_bundle.pywith the arithmetic in the commit body. - Commit rhythm per batch: pin artifacts → author packages → record checklist rows → re-measure bundle expectations. R2 publishes land under
raw/nz/…(#175) in a separate credentialedpublish-rawcommit per batch — designed-red onraw_artifacts_have_r2until it lands; the credential is a PolicyEngine Cloudflare account membership +wrangler login. - Multi-geography tables (#1, #3, #5, #14) use the whole-table orientation: one record set per publisher category,
rows:carry geometry only, semantics inshared_filters/shared_constraints; declaresheets:on multi-sheet workbooks to cap source-cell cost. - A country regression test file
tests/test_nz_targets.py(Belgium'stest_belgium_targets.pyprecedent), grown per batch. - Gate per package:
chronicle validate-package→chronicle build-suite→reports/agent_acceptance.jsonall-green (watch formissing_raw_r2_link,missing_concept_alignment_evidence,selected_row_only_source_parse,missing_expected_constraints).
Acceptance
- A consumer artifact built from these packages resolves every one of the 15 package families — each margin matched by period, geography level, and unit against the plan's margin table; the 16-margin accounting closes with the minimum-wage rules-side note.
- Every fact traces to publisher bytes: manifest
source_url+sha256+raw/nz/…R2 URI, cell-geometry guards,concept_evidence_urlper measure. - Deterministic checks green (source-package validation, source-cell preservation, consumer-contract validation, raw-facts boundary validation); both judges green.
- No PolicyEngine-computed values enter the store (facts-only ADR): no uprating, no rate×count, no cross-source composition. Anything computed is a populace-side declaration.
Out of scope
Wave-2 take-up/validation packages (#177); local geography (#178); the nz_national profile (forthcoming issue); all populace-side consumption (tracked under microcosm#343).
Sizing
15 packages ≈ 5 PRs: (1) stats_nz demographic anchor — #1, #2, #15 first, because calibration initializes design weights at destination population scale; (2) census structure — #3–#5, establishing the hub-CSV and ADE-curated-extract patterns; (3) IRD — #6–#9, establishing the NZ tax-year convention in one review; (4) MSD — #10–#13, carrying the W&I-region ruling and the one PDF package; (5) MBIE rents alone — #14, the big multi-geography whole-table.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the .github/chronicle-agents.yml lane, the package inventory, and the existing source-package precedents. Run chronicle validate-package and inspect tests/test_chronicle_bundle.py plus the planned tests/test_nz_targets.py while implementing one sized batch. Done means all 15 package families resolve, publisher provenance and geometry are preserved, and the validation judges are green.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100