PolicyEngine / PolicyEngine/microcosm

Stage the release builder: bound the dense reconcile working set (~0.5TB measured) via cache spill + checkpointed passes

Open
#463 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

What

Apply the #406 staging discipline to the release builder: bound tools/build_us_fiscal_refresh_release.py's live working set so dense builds fit a ~256GB box again, instead of renting RAM to cover ~half a terabyte of accreted state.

Evidence (Build M dense reruns, 2026-07-19/20, n2-highmem-32 → n2-highmem-80)

#461 (merged) eliminated the reference-lifetime pathology — post-fix runs show RSS returning between families (43GB → 10GB early; slope 0.4GB/min vs 2.5 pre-fix) and per-family sweeps visibly reclaiming swap. What remains is legitimate live state, measured on the full dense pool (RID …20260720T000722Z):

  • First crest ≈ 429GB total (RAM 251 + swap 178) at the eligibility/full-pool high-water mark — survived, then receded to ~400GB through the JCT reform families.
  • Second crest ≈ 536GB (251 + 285) in the SSI take-up reconcile passes, oscillating 240–310GB of swap as passes rematerialize and sweep.
  • Consequence: once the active arrays exceeded physical RAM, throughput collapsed — 6 hours with zero release-log writes, ~0.93 effective cores on 32 vCPU, continuous two-way paging (4–12MB/s si+so). The run was correct but pacing toward days; resized to n2-highmem-80 (640GB) to finish ($5.2/hr vs ~$2.1).

Shape of the fix (the #406 pattern, already proven on the base builder)

  • Spill materialized families through the existing target_materialization_cache instead of holding the full target frame + all family outputs resident: materialize → persist → drop frames; the compile/solve step streams from cache.
  • Fresh-process (or checkpoint-bounded) reconcile passes: each SSI reconcile pass rematerializes everything; give each pass the #406 stage treatment — lossless frame checkpoints between passes so pass N's memory dies with it. The checkpoint/round-trip machinery from #406 is directly reusable.
  • Per-stage RSS/wall profiling to a stage_profile.json twin, so the working-set budget is a regression-tested number rather than a rediscovery.
  • Acceptance: dense full-pool build completes on n2-highmem-32 (256GB, no swap dependence) with bit-identical exports vs the monolith on a pinned subsample — the #406 equivalence-proof pattern.

Prior art: codex/jct-materialization-memory-20260617 (June branch, same instinct pre-#461); measurement trail on #456. Slots into the Build N epic (#449) as the dense-lane front item alongside the CD surface.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tools/build_us_fiscal_refresh_release.py and the #406 staging discipline, especially target_materialization_cache and the checkpoint/round-trip machinery. Compare the prior-art branch codex/jct-materialization-memory-20260617 and measurement trail on #456, then add per-stage measurements to a stage_profile.json twin. Done means the dense full-pool build fits n2-highmem-32 without swap and produces bit-identical exports on a pinned subsample.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, performance, release
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.