PolicyEngine / PolicyEngine/policyengine.py

examples/us_budgetary_impact.py consumes excessive memory (peak ~7.5 GB RSS / 82 GB VMS)

Open
#328 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7
Forks
9
Avg merge
15h 51m
Merged PRs (30d)
9

Description

Reported in the discussion on #264 (JOSS paper PR).

When run on the joss-paper branch (4.2.1), examples/us_budgetary_impact.py exhibits very high memory pressure during the baseline + reform simulations on the enhanced CPS 2024 dataset (2026, ~101k people / 41k households):

  • Vahid: OOM-killed at peak ~30 GB RSS on a Mac (rc=137).
  • Anthony (instrumented run): peak 7.48 GiB RSS, 82.35 GiB VMS; baseline_sim.run() 546.6s wall, reform_sim.run() 596.5s wall; VSCode crashed mid-run from memory pressure.

This makes the canonical US population example impractical for normal local development and for JOSS reviewers attempting to reproduce results.

Suggested next steps:

  • Profile baseline_sim.run() and reform_sim.run() to find the dominant allocations (likely in policyengine-us core, but worth confirming).
  • Determine whether intermediate arrays can be released between runs or whether the two simulations can share underlying data.
  • Document peak memory requirement in the example header until the footprint is reduced.

Related: #264

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/us_budgetary_impact.py and reproduce the baseline_sim.run() and reform_sim.run() calls on the enhanced CPS 2024 dataset. Profile both runs to identify dominant allocations and determine whether intermediate data can be released or shared. Done means reducing the memory footprint or documenting the peak requirement in the example header.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.