PolicyEngine / PolicyEngine/policyengine-core
int32 / float32 default dtypes risk silent overflow and precision loss
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 22
- Forks
- 30
- Avg merge
- 14h 33m
- Merged PRs (30d)
- 7
Description
Problem
policyengine_core/variables/config.py:16-29 sets the default dtypes for integer / float variables to int32 / float32:
int32caps at 2.147e9. National-level aggregates (e.g. total SSA outlays, total payroll) overflow silently above that ceiling.float32carries ~7 decimal digits of precision. Incomes above ~$10M lose dollars; dollar-level tax calculations on multi-million-dollar incomes lose cents. 25_000_000 and 25_000_001 are indistinguishable in float32.
Every country package inherits this precision limit. The related bug H6 (assert_near also uses float32, being fixed in a separate PR) means the test suite cannot currently catch dollar-level regressions on large values.
Why this isn't a drop-in fix
Changing the defaults to int64 / float64 would make pre-existing H5 datasets incompatible with new readers and vice versa. Specifically:
Holder.put_in_cache/set_inputcurrently casts incoming arrays to the variable's dtype.Dataset.savewrites arrays as-is into H5 — existing datasets containfloat32/int32arrays.Simulation.calculatereturnsholder.default_array()which is shaped by the dtype.
A naive swap would:
- Read existing H5 as float32, then upcast on every read (memory overhead).
- Read back datasets written by the new code as float64, but
tax_benefit_systemformulas assume the dtype matchesvariable.dtype. - Invalidate every on-disk cache (each variable's
.npyfile stores the array with its original dtype).
Proposed migration plan
- Add an opt-in flag on
TaxBenefitSystem— e.g.use_extended_precision: bool = False— that forces all newly-built variables toint64/float64. - For one release, emit a
DeprecationWarningwhen a country package constructs a variable with the defaultint32/float32dtype. - Bump that default to
int64/float64the release after. - Provide a migration utility:
policyengine-core data migrate-dtype <path-to-h5>that promotes arrays in-place. - Update country-package CI to validate that their data still round-trips through the new defaults.
References
Identified in the 2026-04 bug hunt (finding H5). Related: H6 (assert_near float32 downcast) is being fixed separately in a non-breaking way.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with policyengine_core/variables/config.py:16-29, then trace Holder.put_in_cache, set_input, Dataset.save, and Simulation.calculate to understand dtype propagation. Review the proposed TaxBenefitSystem flag, deprecation path, migration utility, and country-package CI round-trip requirement. Done means the migration preserves compatibility while preventing silent overflow and precision loss under the new defaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cli, data
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100