opensanctions / opensanctions/poliloom
Migrate to content-addressable storage for archived sources
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 22
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Problem
Content is stored at {YYYY}/{MM}/{DD}/{url_hash}.{ext} where url_hash = SHA256(url)[:16]. This has two issues:
- Collision: Same URL fetched twice on the same day writes to the same file path, silently overwriting content. If the page changed between fetches, the first source's archived content is lost.
- Wrong abstraction: The hash is based on the URL, not the content — it's neither deduplication nor collision-safe.
Solution
Switch to content-addressable storage (CAS), like Git's object store:
- Hash the actual content (SHA-256 of file bytes), not the URL
- Path structure:
{hash[:2]}/{hash[2:4]}/{hash}.{ext}(2 levels of 2-char dirs) - Natural deduplication: identical content stored once, different content never collides
- Idempotent writes: check if file exists before writing
Changes needed
- Drop
url_hashcolumn entirely (the URL itself is already stored) - Add
content_hashandhtml_content_hashcolumns (set after fetch, when content is available) - Replace
path_rootproperty withcas_path()deriving paths from content hashes - Replace
save_archived_content/read_archived_contentwith CAS equivalents - Alembic migration including file migration from old paths to new CAS paths
- Update API schema, frontend types, and tests
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating path_root, save_archived_content, read_archived_content, the archive model, Alembic migrations, API schema, frontend types, and related tests. Trace how hashes and archived files are currently created and read before planning the migration. Done means CAS paths and content hashes are used consistently, old files are migrated, and the API, frontend types, and tests cover the new behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, databases, frontend
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100