PolicyEngine / PolicyEngine/policyengine-sim-api
Segmented-national follow-ups from the #637 review
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- Avg merge
- 10h 29m
- Merged PRs (30d)
- 7
Description
Remaining items from the PR #637 code review, deliberately deferred:
Efficiency (review finding 9):
- Lazy-import the reduce chain inside the runner so cold
run_simulationcontainers don't pay ~19s importing pandas/microdf/policyengine.core before routing (regional/UK requests pay it today and never use it) - Add the
POLICYENGINE_SKIP_COUNTRY_IMPORTSsetdefault guard tosegmented_national_reduce.py— a barepytest tests/test_segmented_national*.pyprocess currently imports both country models (~80s) to run pure-mechanics unit tests
Robustness (review finding 10):
- National requests with a custom
data/data_versioncurrently fan out; small custom datasets make every child fail the >100-household assertion where a monolithic run succeeded. Either add custom-data requests to the monolithic fallback, or make the partition dataset-aware.
DRY leftovers (finding 14, partially addressed):
- The poll/backoff loops in
budget_window_scheduler.pyandsegmented_national.pyremain separate (payload building + backoff step now sharesrc/modal/fanout.py); extracting a shared poll-with-backoff helper is still open, as is consolidating the three in-repo Modal fakes (test_budget_window_scheduler.pymocks vstest_segmented_national.pyfakes) into a conftest fixture
Validation debt:
-
include_cliffssegmentation: verify cliff-variable reconstruction through the reduce, then lift the monolithic fallback - Labor-supply-response segmentation: requires recomputing national
household_income_decileat reduce time (the transported column is region-group-relative) and validating against monolithic — see theLSR_PARAMETER_PREFIXcomment insegmented_national.py - Budget-window children remain pinned
segmented=false; enabling nested fan-out (years x 21 containers) needs capacity planning against the 100/300 pools
Related: #638 (deprecating the region:"us" national spelling).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with segmented_national.py and segmented_national_reduce.py, then review budget_window_scheduler.py, src/modal/fanout.py, and the related tests named in the issue. Run the segmented-national tests first and validate each deferred item against monolithic results, custom-data behavior, import time, and pool capacity; the work is done when the listed review and validation gaps are resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- backend, data, performance, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100