PolicyEngine / PolicyEngine/policyengine.py
examples/us_budgetary_impact.py consumes excessive memory (peak ~7.5 GB RSS / 82 GB VMS)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7
- Forks
- 9
- Avg merge
- 15h 51m
- Merged PRs (30d)
- 9
Description
Reported in the discussion on #264 (JOSS paper PR).
When run on the joss-paper branch (4.2.1), examples/us_budgetary_impact.py exhibits very high memory pressure during the baseline + reform simulations on the enhanced CPS 2024 dataset (2026, ~101k people / 41k households):
- Vahid: OOM-killed at peak ~30 GB RSS on a Mac (rc=137).
- Anthony (instrumented run): peak 7.48 GiB RSS, 82.35 GiB VMS;
baseline_sim.run()546.6s wall,reform_sim.run()596.5s wall; VSCode crashed mid-run from memory pressure.
This makes the canonical US population example impractical for normal local development and for JOSS reviewers attempting to reproduce results.
Suggested next steps:
- Profile
baseline_sim.run()andreform_sim.run()to find the dominant allocations (likely inpolicyengine-uscore, but worth confirming). - Determine whether intermediate arrays can be released between runs or whether the two simulations can share underlying data.
- Document peak memory requirement in the example header until the footprint is reduced.
Related: #264
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with examples/us_budgetary_impact.py and reproduce the baseline_sim.run() and reform_sim.run() calls on the enhanced CPS 2024 dataset. Profile both runs to identify dominant allocations and determine whether intermediate data can be released or shared. Done means reducing the memory footprint or documenting the peak requirement in the example header.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100