opensanctions / opensanctions/poliloom

Migrate to content-addressable storage for archived sources

Open
#145 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

loom
Dominant language
Python
Stars
22
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Problem

Content is stored at {YYYY}/{MM}/{DD}/{url_hash}.{ext} where url_hash = SHA256(url)[:16]. This has two issues:

  1. Collision: Same URL fetched twice on the same day writes to the same file path, silently overwriting content. If the page changed between fetches, the first source's archived content is lost.
  2. Wrong abstraction: The hash is based on the URL, not the content — it's neither deduplication nor collision-safe.

Solution

Switch to content-addressable storage (CAS), like Git's object store:

  • Hash the actual content (SHA-256 of file bytes), not the URL
  • Path structure: {hash[:2]}/{hash[2:4]}/{hash}.{ext} (2 levels of 2-char dirs)
  • Natural deduplication: identical content stored once, different content never collides
  • Idempotent writes: check if file exists before writing

Changes needed

  • Drop url_hash column entirely (the URL itself is already stored)
  • Add content_hash and html_content_hash columns (set after fetch, when content is available)
  • Replace path_root property with cas_path() deriving paths from content hashes
  • Replace save_archived_content/read_archived_content with CAS equivalents
  • Alembic migration including file migration from old paths to new CAS paths
  • Update API schema, frontend types, and tests

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating path_root, save_archived_content, read_archived_content, the archive model, Alembic migrations, API schema, frontend types, and related tests. Trace how hashes and archived files are currently created and read before planning the migration. Done means CAS paths and content hashes are used consistently, old files are migrated, and the API, frontend types, and tests cover the new behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, databases, frontend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.