PolicyEngine / PolicyEngine/microcosm
Stage the release builder: bound the dense reconcile working set (~0.5TB measured) via cache spill + checkpointed passes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
What
Apply the #406 staging discipline to the release builder: bound tools/build_us_fiscal_refresh_release.py's live working set so dense builds fit a ~256GB box again, instead of renting RAM to cover ~half a terabyte of accreted state.
Evidence (Build M dense reruns, 2026-07-19/20, n2-highmem-32 → n2-highmem-80)
#461 (merged) eliminated the reference-lifetime pathology — post-fix runs show RSS returning between families (43GB → 10GB early; slope 0.4GB/min vs 2.5 pre-fix) and per-family sweeps visibly reclaiming swap. What remains is legitimate live state, measured on the full dense pool (RID …20260720T000722Z):
- First crest ≈ 429GB total (RAM 251 + swap 178) at the eligibility/full-pool high-water mark — survived, then receded to ~400GB through the JCT reform families.
- Second crest ≈ 536GB (251 + 285) in the SSI take-up reconcile passes, oscillating 240–310GB of swap as passes rematerialize and sweep.
- Consequence: once the active arrays exceeded physical RAM, throughput collapsed — 6 hours with zero release-log writes, ~0.93 effective cores on 32 vCPU, continuous two-way paging (
4–12MB/s si+so). The run was correct but pacing toward days; resized to n2-highmem-80 (640GB) to finish ($5.2/hr vs ~$2.1).
Shape of the fix (the #406 pattern, already proven on the base builder)
- Spill materialized families through the existing
target_materialization_cacheinstead of holding the full target frame + all family outputs resident: materialize → persist → drop frames; the compile/solve step streams from cache. - Fresh-process (or checkpoint-bounded) reconcile passes: each SSI reconcile pass rematerializes everything; give each pass the #406 stage treatment — lossless frame checkpoints between passes so pass N's memory dies with it. The checkpoint/round-trip machinery from #406 is directly reusable.
- Per-stage RSS/wall profiling to a
stage_profile.jsontwin, so the working-set budget is a regression-tested number rather than a rediscovery. - Acceptance: dense full-pool build completes on n2-highmem-32 (256GB, no swap dependence) with bit-identical exports vs the monolith on a pinned subsample — the #406 equivalence-proof pattern.
Prior art: codex/jct-materialization-memory-20260617 (June branch, same instinct pre-#461); measurement trail on #456. Slots into the Build N epic (#449) as the dense-lane front item alongside the CD surface.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tools/build_us_fiscal_refresh_release.py and the #406 staging discipline, especially target_materialization_cache and the checkpoint/round-trip machinery. Compare the prior-art branch codex/jct-materialization-memory-20260617 and measurement trail on #456, then add per-stage measurements to a stage_profile.json twin. Done means the dense full-pool build fits n2-highmem-32 without swap and produces bit-identical exports on a pinned subsample.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, performance, release
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100