PolicyEngine / PolicyEngine/policyengine-scorecard
Intern claim provenance: publications + reforms side tables (~20 MB of inline JSON duplication)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 6d 12h
- Merged PRs (30d)
- 32
Description
Measured on the 2026-08-28 build (58,473 claims, 57.7 MB): external_scores.publication is 15.4 MB of JSON with only 8,656 distinct dicts, and reform_json is 4.9 MB with only 289 distinct values — every claim row re-carries its full provenance inline. That is ~20 MB of duplication (35% of the file), and it is the dominant scaling term for the upcoming populations (the OBR Policy Measures Database long tail alone is ~24k rows sharing a handful of publications).
Proposal — intern both, following the baselines-registry template:
publications(publication_id, publication), content-addressed (sha256(canonical)[:16], same construction asbaseline_key).reforms(reform_key, reform_json)—reform_keyis already the hash of exactly that JSON, so this is pure normalization; the column already exists and is indexed onexternal_scores.- The
comparisonsview re-exposes both columns via joins, so view readers are untouched; the handful of base-table readers (export_populations, the campaign claim-matcher predicate, the reform-validation registry predicates, three tests) join the side tables. pe_exhibits.reform_jsonstays inline deliberately: 61 KB total, 9 distinct values, and itsreform_keynamespace is campaign policy slugs, notReformRef.key()hashes — interning it would conflate two key namespaces for no material win.- Migration converts pre-interning files in place (after the legacy baseline backfill, which reads the inline
reform_jsonon pre-#13 files);build_dbgains an orphan gate so a rawSCORES_SQLwriter that skips the provenance rows fails the build.
Effect: 57.7 MB → 39.9 MB (−31%) on today's catalog, unchanged feeds (byte-identical populations.json), unchanged claim ids, deterministic double-build.
Not in scope: conditions (8.1 MB) — per-claim identity, low dedup value; the Supabase object shrinks from ~6 MB to ~4.3 MB gzipped as a side effect.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the baselines-registry template, then trace build_db and the raw SCORES_SQL writer to understand provenance-row creation and the orphan gate. Check export_populations, the campaign claim-matcher and reform-validation registry predicates, the comparisons view, and the three affected tests; done means side-table reads work, feeds and claim IDs remain unchanged, and double builds are deterministic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- backend, databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100