PolicyEngine / PolicyEngine/policyengine-scorecard

Intern claim provenance: publications + reforms side tables (~20 MB of inline JSON duplication)

Open
#112 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
0
Avg merge
6d 12h
Merged PRs (30d)
32

Description

Measured on the 2026-08-28 build (58,473 claims, 57.7 MB): external_scores.publication is 15.4 MB of JSON with only 8,656 distinct dicts, and reform_json is 4.9 MB with only 289 distinct values — every claim row re-carries its full provenance inline. That is ~20 MB of duplication (35% of the file), and it is the dominant scaling term for the upcoming populations (the OBR Policy Measures Database long tail alone is ~24k rows sharing a handful of publications).

Proposal — intern both, following the baselines-registry template:

  • publications(publication_id, publication), content-addressed (sha256(canonical)[:16], same construction as baseline_key).
  • reforms(reform_key, reform_json)reform_key is already the hash of exactly that JSON, so this is pure normalization; the column already exists and is indexed on external_scores.
  • The comparisons view re-exposes both columns via joins, so view readers are untouched; the handful of base-table readers (export_populations, the campaign claim-matcher predicate, the reform-validation registry predicates, three tests) join the side tables.
  • pe_exhibits.reform_json stays inline deliberately: 61 KB total, 9 distinct values, and its reform_key namespace is campaign policy slugs, not ReformRef.key() hashes — interning it would conflate two key namespaces for no material win.
  • Migration converts pre-interning files in place (after the legacy baseline backfill, which reads the inline reform_json on pre-#13 files); build_db gains an orphan gate so a raw SCORES_SQL writer that skips the provenance rows fails the build.

Effect: 57.7 MB → 39.9 MB (−31%) on today's catalog, unchanged feeds (byte-identical populations.json), unchanged claim ids, deterministic double-build.

Not in scope: conditions (8.1 MB) — per-claim identity, low dedup value; the Supabase object shrinks from ~6 MB to ~4.3 MB gzipped as a side effect.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the baselines-registry template, then trace build_db and the raw SCORES_SQL writer to understand provenance-row creation and the orphan gate. Check export_populations, the campaign claim-matcher and reform-validation registry predicates, the comparisons view, and the three affected tests; done means side-table reads work, feeds and claim IDs remain unchanged, and double builds are deterministic.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
backend, databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.