PolicyEngine / PolicyEngine/policyengine-core

int32 / float32 default dtypes risk silent overflow and precision loss

Open
#467 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
22
Forks
30
Avg merge
14h 33m
Merged PRs (30d)
7

Description

Problem

policyengine_core/variables/config.py:16-29 sets the default dtypes for integer / float variables to int32 / float32:

  • int32 caps at 2.147e9. National-level aggregates (e.g. total SSA outlays, total payroll) overflow silently above that ceiling.
  • float32 carries ~7 decimal digits of precision. Incomes above ~$10M lose dollars; dollar-level tax calculations on multi-million-dollar incomes lose cents. 25_000_000 and 25_000_001 are indistinguishable in float32.

Every country package inherits this precision limit. The related bug H6 (assert_near also uses float32, being fixed in a separate PR) means the test suite cannot currently catch dollar-level regressions on large values.

Why this isn't a drop-in fix

Changing the defaults to int64 / float64 would make pre-existing H5 datasets incompatible with new readers and vice versa. Specifically:

  • Holder.put_in_cache / set_input currently casts incoming arrays to the variable's dtype.
  • Dataset.save writes arrays as-is into H5 — existing datasets contain float32 / int32 arrays.
  • Simulation.calculate returns holder.default_array() which is shaped by the dtype.

A naive swap would:

  1. Read existing H5 as float32, then upcast on every read (memory overhead).
  2. Read back datasets written by the new code as float64, but tax_benefit_system formulas assume the dtype matches variable.dtype.
  3. Invalidate every on-disk cache (each variable's .npy file stores the array with its original dtype).

Proposed migration plan

  1. Add an opt-in flag on TaxBenefitSystem — e.g. use_extended_precision: bool = False — that forces all newly-built variables to int64 / float64.
  2. For one release, emit a DeprecationWarning when a country package constructs a variable with the default int32/float32 dtype.
  3. Bump that default to int64/float64 the release after.
  4. Provide a migration utility: policyengine-core data migrate-dtype <path-to-h5> that promotes arrays in-place.
  5. Update country-package CI to validate that their data still round-trips through the new defaults.

References

Identified in the 2026-04 bug hunt (finding H5). Related: H6 (assert_near float32 downcast) is being fixed separately in a non-breaking way.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with policyengine_core/variables/config.py:16-29, then trace Holder.put_in_cache, set_input, Dataset.save, and Simulation.calculate to understand dtype propagation. Review the proposed TaxBenefitSystem flag, deprecation path, migration utility, and country-package CI round-trip requirement. Done means the migration preserves compatibility while preventing silent overflow and precision loss under the new defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli, data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.