PolicyEngine / PolicyEngine/policyengine-scorecard

History slimming runbook: 757 MB of dead scorecard.db blobs, execute at the repo-public flip (per #74)

Open
#113 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
0
Avg merge
6d 12h
Merged PRs (30d)
32

Description

PR #74 took data/scorecard.db out of git going forward and recorded: "History slimming of the old blobs is deliberately deferred to just before the repo-public flip." This issue makes that deferred intention trackable and carries the runbook + measurements so the flip-day execution is mechanical.

What history carries today (measured 2026-08-28):

  • 19 distinct data/scorecard.db blobs on main, 757 MB raw input (SQLite binaries barely delta-compress; the shared pack is ~275 MB — this is most of every clone).
  • Second bleed, ongoing: data/comparison.json + its byte-identical copy app/public/data/comparison.json at ~22 MB × 2 per platform-grid refresh (text deltas pack far better than SQLite, but it's the next-largest family; worth deduplicating the app copy at the same time or before).

Runbook (execute just before the repo-public flip, per #74):

  1. Freeze merges; drain or park the PR queue (a rewrite force-rebases every open PR — there are 17 open today, which is exactly why not now).
  2. Fresh clone → git filter-repo --path data/scorecard.db --invert-paths (the path is untracked at tip, so this is a pure history prune — nothing at HEAD changes).
  3. Optionally add --strip-blobs-bigger-than for historical comparison.json versions if the app copy has been deduplicated by then (the path is still tracked, so a bare --invert-paths on it would delete the live file — don't).
  4. Force-push main + tags; contributors re-clone; note in the README that pre-rewrite SHAs are dead.
  5. Expected result: pack ~275 MB → well under 100 MB.

Needs an explicit go from @MaxGhenis at execution time — it rewrites every SHA after the first DB commit.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read PR #74 and this runbook before acting; the relevant paths are data/scorecard.db and, if deduplicated first, data/comparison.json and app/public/data/comparison.json. At the repo-public flip, obtain @MaxGhenis's approval, freeze merges, and run the documented git filter-repo procedure from a fresh clone. Done means the rewritten main and tags are pushed, contributors are warned about dead SHAs, and the pack is under 100 MB.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, github
Domain
devops, release
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.