PolicyEngine / PolicyEngine/policyengine-core
ID variables produce fractional values when mapped between non-hierarchical entities
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 22
- Forks
- 30
- Avg merge
- 14h 33m
- Merged PRs (30d)
- 7
Description
Problem Description
When mapping ID variables (like household_id, spm_unit_id) between non-hierarchical entities using calculate(..., map_to=...), PolicyEngine Core inappropriately averages these values, producing nonsensical fractional IDs.
Minimal Reproducible Example
from policyengine_us import Microsimulation
import pandas as pd
sim = Microsimulation(dataset='hf://policyengine/policyengine-us-data/enhanced_cps_2024.h5')
# Map household_id to tax_unit level
household_ids_tax_unit = sim.calculate('household_id', map_to='tax_unit')
# Check for fractional values (IDs should always be integers)
fractional_ids = household_ids_tax_unit.values[household_ids_tax_unit.values % 1 != 0]
print(f"Found {len(fractional_ids)} fractional household IDs")
print(f"Examples: {fractional_ids[:5]}")
# Output: [218. 153.5 153.5 172.5 172.5]
Root Cause
In policyengine_core/simulations/simulation.py, the map_result method handles mapping between non-hierarchical group entities (e.g., household → tax_unit) by:
- First mapping from source to person using
how="mean"(averaging) - Then mapping from person to target using
how="sum"(summing)
This is mathematically inappropriate for ID fields, which are categorical identifiers, not numeric quantities that should be averaged or summed.
Impact
- Produces invalid ID values that break referential integrity
- Can cause silent bugs in downstream analysis
- Affects any code that relies on ID mapping between non-hierarchical entities
Proposed Solutions
- Short-term: Add a warning when mapping variables with "_id" suffix between non-hierarchical entities
- Medium-term: Add a variable attribute to mark categorical/ID variables that should not be aggregated mathematically
- Long-term: Implement proper ID mapping logic that preserves the most common ID or uses a different strategy appropriate for categorical data
Affected Variables
Testing shows at least these ID variables produce fractional values when mapped to tax_unit:
household_idspm_unit_id
Environment
- policyengine-core version: 3.20.0
- policyengine-us version: 1.399.1
- Python version: 3.13
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the fractional IDs with the minimal example, then inspect policyengine_core/simulations/simulation.py, especially map_result and its person-to-entity mappings. Compare the current mean and sum behavior for household_id and spm_unit_id, and clarify which proposed ID-handling strategy is expected. Done means mapped ID variables no longer produce invalid fractional identifiers and the behavior is covered by an appropriate test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100