PolicyEngine / PolicyEngine/microcosm

Add the BE-SILC source stage with licence guardrails and a private artifact repo

Open
#262 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Part of #259 (populace-be epic).

What

The BE base microdata stage: BE-SILC (Statbel national SILC preferred over the Eurostat UDB where variable richness matters) mapped into Frame entity tables — plus the licence guardrails, which land before any upload code exists.

Licence posture (blocking, first PR of this issue)

BE-SILC scientific-use access is restricted (Statbel/Eurostat terms; institutional access holder TBD — scoping discussions from IMA 2026). Mirror the UK machinery exactly:

  • Private HF dataset repo (e.g. policyengine/populace-be-private) as the only microdata destination.
  • A destinations allowlist module + AST-scan test that fails CI if any upload call targets a repo outside the allowlist (pattern: policyengine-uk-data policyengine_uk_data/utils/hf_destinations.py + tests/test_hf_destinations.py).
  • Documented public/private split: manifests, calibration diagnostics, source coverage, reform validation are public; microdata, and any per-record output, never.
  • No individual-level records in logs or CI output; aggregates only.

Source stage

  • SILC household/personal registers → Frame person + household tables with explicit id linkage; document the survey-year vs income-reference-year lag (SILC year N carries year N−1 incomes) — the build's period bookkeeping and the target profiles must agree on basis period.
  • Entity derivation for BE units per the adapter's entity_schema() (individual-centric PIT with household elements; benefit units) — the adapter issue owns the schema, this issue owns populating it from SILC.
  • Region (NUTS1) comes from SILC; commune-level geography is assigned by the clone/geography spine stage (separate calibration issue), not claimed from SILC.
  • Weights: SILC design weights enter as DESIGN-kind weights with mass conservation asserted.

Acceptance

  • Raw SILC → Frame build runs end-to-end in the licensed environment.
  • Destination-guard tests in CI from the first PR that touches upload paths.
  • Stage is expressed as a spec entry referencing named operators (see the spec-schema issue), not a BE Python module.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the UK reference files policyengine_uk_data/utils/hf_destinations.py and tests/test_hf_destinations.py, then review #259 and the spec-schema issue. Check the adapter's entity_schema() and named operators before defining the BE-SILC spec entry. Done means the licensed raw-to-Frame build runs, destination guards pass in CI, and only permitted aggregate/public outputs are exposed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.