PolicyEngine / PolicyEngine/microcosm

Release reform-validation tail derives UNKNOWN county for records with valid county_fips (blocks manifest)

Open
#367 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Reform-validation tail derives county = UNKNOWN for records whose exported artifact carries valid county_fips

The release tool's reform-validation tail (tools/build_us_fiscal_refresh_release.py, the batched zero-variable-reform simulation built from the frame) crashes at the 2025 period with ParameterNotFoundError: 'gov.local.md.flat_rate.UNKNOWN'even though the exported artifact's MD households all carry valid county_fips. This blocks release_manifest.json (rc=1) after all certification gates + the export-mass gate + the H5 have already been written.

Root cause of the raise itself is a policyengine-us fragility (md_withheld_income_tax has no UNKNOWN-county guard — see cross-linked issue). But the reason an UNKNOWN county appears at all is harness-side: the exported dataset is geographically clean, so the batched reform-validation construction is losing/mis-deriving county for some MD record.

Three-way clean-artifact evidence (Build I sparse, rmloss100)
  1. Exported H5 via the tail's own loader (USSingleYearDataset(file_path=…)Microsimulation, _write_reform_validation line ~4691): all 1,091 MD households map to valid MD counties at both 2024 and 2025; ZERO UNKNOWN counties anywhere in the 57,240 households; md_withheld_income_tax@2025 computes cleanly ($28.13B). The H5 persists valid county_fips for all 24 MD jurisdictions (2400124510).
  2. Frame path is clean: the target-frame checkpoint carries county_fips/state_fips/place_fips; _dataset_from_frame preserves them; the full-frame reform-validation dataset also computes md_withheld with 0 UNKNOWN.
  3. Full-frame reforms are clean: neutralizing salt_deduction, interest_deduction, and qualified_business_income_deduction and computing income_tax on the full frame all succeed. The crash appears only in the batched construction (_reform_household_income_tax / _batched_reform_validation_simulate_factory_from_frame, ~lines 2357–2431: _select_households_by_position(base_frame, positions)_dataset_from_frame(batch_frame)Microsimulation(dataset=reformed_dataset, reform=reform) at 2025).

So a per-batch frame slice appears to drop or fail to carry county_fips (or its downstream county) for at least one MD household, which then trips the unguarded MD flat-rate lookup.

The live default is more exposed, not less

The current live/default artifact (c2065b64) persists no county_fips at all (only state_fips), so its reform-validation would derive UNKNOWN for every MD household and hit this row immediately if the tail reached it. Build I actually persists correct county geography; earlier runs' tails were interrupted before reaching this row, which is why it had not surfaced.

Traceback (verbatim)
File ".../salt_deduction.py", line 16, in formula
File ".../state_and_local_sales_or_income_tax.py", line 13, in formula
File ".../gov/states/md/tax/income/md_withheld_income_tax.py", line 28, in formula
    flat_rate_withheld = p_local.flat_rate[safe_county] * reduced_agi
...
ParameterNotFoundError: The parameter 'gov.local.md.flat_rate.UNKNOWN' was not found in the 2025-01-01 tax and benefit system.
Workaround (used for Build I)

--skip-reform-validation completes the release and writes release_manifest.json; the reform-validation report is optional and all certification output (gates, export-mass parity, H5) is produced before the tail. The dataset is unaffected.

Suggested fix

Ensure the batched reform-validation frame slices carry county_fips/county end-to-end (so the reformed dataset matches the exported artifact's geography), and/or adopt the policyengine-us UNKNOWN-county guard (cross-linked issue) so the validation report is robust to any dataset lacking county detail (e.g. the current PUF-derived default).

Cross-reference (root-cause formula fragility): https://github.com/PolicyEngine/policyengine-us/issues/8975

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tools/build_us_fiscal_refresh_release.py around _reform_household_income_tax and _batched_reform_validation_simulate_factory_from_frame, then inspect _select_households_by_position and _dataset_from_frame. Run the reform-validation tail on the clean Build I frame and compare county_fips/county through each batch slice. Done means the batched validation preserves valid Maryland county geography and completes without an UNKNOWN county lookup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
release
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.