PolicyEngine / PolicyEngine/policyengine-core

ID variables produce fractional values when mapped between non-hierarchical entities

Open
#400 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
22
Forks
30
Avg merge
14h 33m
Merged PRs (30d)
7

Description

Problem Description

When mapping ID variables (like household_id, spm_unit_id) between non-hierarchical entities using calculate(..., map_to=...), PolicyEngine Core inappropriately averages these values, producing nonsensical fractional IDs.

Minimal Reproducible Example

from policyengine_us import Microsimulation
import pandas as pd

sim = Microsimulation(dataset='hf://policyengine/policyengine-us-data/enhanced_cps_2024.h5')

# Map household_id to tax_unit level
household_ids_tax_unit = sim.calculate('household_id', map_to='tax_unit')

# Check for fractional values (IDs should always be integers)
fractional_ids = household_ids_tax_unit.values[household_ids_tax_unit.values % 1 != 0]
print(f"Found {len(fractional_ids)} fractional household IDs")
print(f"Examples: {fractional_ids[:5]}")
# Output: [218.  153.5 153.5 172.5 172.5]

Root Cause

In policyengine_core/simulations/simulation.py, the map_result method handles mapping between non-hierarchical group entities (e.g., household → tax_unit) by:

  1. First mapping from source to person using how="mean" (averaging)
  2. Then mapping from person to target using how="sum" (summing)

This is mathematically inappropriate for ID fields, which are categorical identifiers, not numeric quantities that should be averaged or summed.

Impact

  • Produces invalid ID values that break referential integrity
  • Can cause silent bugs in downstream analysis
  • Affects any code that relies on ID mapping between non-hierarchical entities

Proposed Solutions

  1. Short-term: Add a warning when mapping variables with "_id" suffix between non-hierarchical entities
  2. Medium-term: Add a variable attribute to mark categorical/ID variables that should not be aggregated mathematically
  3. Long-term: Implement proper ID mapping logic that preserves the most common ID or uses a different strategy appropriate for categorical data

Affected Variables

Testing shows at least these ID variables produce fractional values when mapped to tax_unit:

  • household_id
  • spm_unit_id

Environment

  • policyengine-core version: 3.20.0
  • policyengine-us version: 1.399.1
  • Python version: 3.13

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the fractional IDs with the minimal example, then inspect policyengine_core/simulations/simulation.py, especially map_result and its person-to-entity mappings. Compare the current mean and sum behavior for household_id and spm_unit_id, and clarify which proposed ID-handling strategy is expected. Done means mapped ID variables no longer produce invalid fractional identifiers and the behavior is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.