PolicyEngine / PolicyEngine/microcosm

SYSTEM_REQUIREMENTS.md memory model is stale: attributes the 29 GB calibration peak to a dense-row vstack the compiler no longer does

Open
#383 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation
Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Summary

SYSTEM_REQUIREMENTS.md explains the ~29 GB calibration peak with a mechanism the code no longer has, so its capacity-planning formula should not be trusted until remeasured.

Evidence

SYSTEM_REQUIREMENTS.md:118-128 states:

peak RAM is set by the compile step ... build_constraint_matrix materializes one dense row per target into a Python list and np.vstacks them before compressing to CSR, so peak RAM ≈ 2 × n_targets × n_records × 8 bytes (the row list + the vstack copy)

Current matrix compilation (packages/populace-calibrate/src/populace/calibrate/matrix.py:198-260) materializes one dense row at a time, immediately reduces it to its nonzero indices/data parts, and builds CSR directly — there is no retained dense-row list and no np.vstack copy. The documented 2×-dense formula (and the 300k-record / 6,288-target ≈ 28-29 GB ceiling derived from it) describes the old implementation.

Peak memory today is more plausibly set elsewhere (six pandas entity-table copies at H5 load, full Microsimulation construction during target materialization, CSR + torch tensors during solve) — which matters for machine sizing on Build J-class runs.

Acceptance criteria

  • Remeasure peak RSS of a current-main calibration at production scale (or a scaled measurement with the scaling law stated), and update the SYSTEM_REQUIREMENTS.md formula and table to the observed mechanism.
  • The dense-row/np.vstack explanation is removed or explicitly marked historical.

Cross-links

  • #299 (the OOM history that motivated the document)
  • #285 (production-scale sweep runs that could produce the measurement for free)

Surfaced by an external architecture review (Codex/gpt-5.6-sol, 2026-07-09); citations independently verified.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read SYSTEM_REQUIREMENTS.md:118-128 alongside packages/populace-calibrate/src/populace/calibrate/matrix.py:198-260, then measure peak RSS for a current-main calibration at production scale or with a stated scaling law. Account for the H5 entity-table copies, Microsimulation construction, and CSR plus torch tensors; update the formula and table to match the observed mechanism and remove or mark the dense-row explanation historical.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python, pytorch
Domain
data, documentation, performance
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.