PolicyEngine / PolicyEngine/microcosm
Add the BE-SILC source stage with licence guardrails and a private artifact repo
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Part of #259 (populace-be epic).
What
The BE base microdata stage: BE-SILC (Statbel national SILC preferred over the Eurostat UDB where variable richness matters) mapped into Frame entity tables — plus the licence guardrails, which land before any upload code exists.
Licence posture (blocking, first PR of this issue)
BE-SILC scientific-use access is restricted (Statbel/Eurostat terms; institutional access holder TBD — scoping discussions from IMA 2026). Mirror the UK machinery exactly:
- Private HF dataset repo (e.g.
policyengine/populace-be-private) as the only microdata destination. - A destinations allowlist module + AST-scan test that fails CI if any upload call targets a repo outside the allowlist (pattern: policyengine-uk-data
policyengine_uk_data/utils/hf_destinations.py+tests/test_hf_destinations.py). - Documented public/private split: manifests, calibration diagnostics, source coverage, reform validation are public; microdata, and any per-record output, never.
- No individual-level records in logs or CI output; aggregates only.
Source stage
- SILC household/personal registers → Frame person + household tables with explicit id linkage; document the survey-year vs income-reference-year lag (SILC year N carries year N−1 incomes) — the build's period bookkeeping and the target profiles must agree on basis period.
- Entity derivation for BE units per the adapter's
entity_schema()(individual-centric PIT with household elements; benefit units) — the adapter issue owns the schema, this issue owns populating it from SILC. - Region (NUTS1) comes from SILC; commune-level geography is assigned by the clone/geography spine stage (separate calibration issue), not claimed from SILC.
- Weights: SILC design weights enter as DESIGN-kind weights with mass conservation asserted.
Acceptance
- Raw SILC → Frame build runs end-to-end in the licensed environment.
- Destination-guard tests in CI from the first PR that touches upload paths.
- Stage is expressed as a spec entry referencing named operators (see the spec-schema issue), not a BE Python module.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the UK reference files policyengine_uk_data/utils/hf_destinations.py and tests/test_hf_destinations.py, then review #259 and the spec-schema issue. Check the adapter's entity_schema() and named operators before defining the BE-SILC spec entry. Done means the licensed raw-to-Frame build runs, destination guards pass in CI, and only permitted aggregate/public outputs are exposed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100