PolicyEngine / PolicyEngine/policyengine-core

Storage keys need a structured (branch, period) scheme: str(period) is lossy and separators collide with branch names

Open
#526 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
22
Forks
30
Avg merge
14h 33m
Merged PRs (30d)
7

Description

#523 / #525 fix the crash where get_known_periods/get_known_branch_periods split storage keys on every colon, but the underlying key scheme has deeper, pre-existing structural problems (surfaced in cross-family review of #525):

  1. str(period) is lossy for day-anchored periods. Keys embed f"{branch_name}:{period}", and Period.__str__ drops the day for e.g. month:2027-11-15:2 / year:2027-11-15:2, so distinct valid periods can collide in the cache and the readers reconstruct date-shifted periods.
  2. Branch names have no enforced grammar. Simulation.get_branch accepts arbitrary names; a branch name containing : breaks split(":", 1) parsing in the readers and in delete.
  3. OnDiskStorage has the same defect class with _ as the separator ({branch}_{period} + unconstrained split("_")): a branch name like my_reform mis-parses, and complex-period colons produce awkward filenames.

The clean fix is structured keys — (branch_name, Period) tuples in memory and an escaped/encoded filename scheme on disk — rather than string round-trips. That is a storage-format refactor deserving its own design + review, hence this issue rather than a rider on #525.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing get_known_periods and get_known_branch_periods, then inspect Simulation.get_branch, delete, OnDiskStorage, and Period. Design structured in-memory keys using (branch_name, Period) and an escaped or encoded on-disk filename scheme. Done means distinct day-anchored periods and branch names containing separators round-trip without collisions or parsing errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.