PolicyEngine / PolicyEngine/microcosm
SYSTEM_REQUIREMENTS.md memory model is stale: attributes the 29 GB calibration peak to a dense-row vstack the compiler no longer does
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Summary
SYSTEM_REQUIREMENTS.md explains the ~29 GB calibration peak with a mechanism the code no longer has, so its capacity-planning formula should not be trusted until remeasured.
Evidence
SYSTEM_REQUIREMENTS.md:118-128 states:
peak RAM is set by the compile step ...
build_constraint_matrixmaterializes one dense row per target into a Python list andnp.vstacks them before compressing to CSR, so peak RAM ≈ 2 × n_targets × n_records × 8 bytes (the row list + the vstack copy)
Current matrix compilation (packages/populace-calibrate/src/populace/calibrate/matrix.py:198-260) materializes one dense row at a time, immediately reduces it to its nonzero indices/data parts, and builds CSR directly — there is no retained dense-row list and no np.vstack copy. The documented 2×-dense formula (and the 300k-record / 6,288-target ≈ 28-29 GB ceiling derived from it) describes the old implementation.
Peak memory today is more plausibly set elsewhere (six pandas entity-table copies at H5 load, full Microsimulation construction during target materialization, CSR + torch tensors during solve) — which matters for machine sizing on Build J-class runs.
Acceptance criteria
- Remeasure peak RSS of a current-main calibration at production scale (or a scaled measurement with the scaling law stated), and update the SYSTEM_REQUIREMENTS.md formula and table to the observed mechanism.
- The dense-row/
np.vstackexplanation is removed or explicitly marked historical.
Cross-links
- #299 (the OOM history that motivated the document)
- #285 (production-scale sweep runs that could produce the measurement for free)
Surfaced by an external architecture review (Codex/gpt-5.6-sol, 2026-07-09); citations independently verified.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read SYSTEM_REQUIREMENTS.md:118-128 alongside packages/populace-calibrate/src/populace/calibrate/matrix.py:198-260, then measure peak RSS for a current-main calibration at production scale or with a stated scaling law. Account for the H5 entity-table copies, Microsimulation construction, and CSR plus torch tensors; update the formula and table to match the observed mechanism and remove or mark the dense-row explanation historical.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python, pytorch
- Domain
- data, documentation, performance
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100