drizzle-team / drizzle-team/drizzle-orm
Feature: delta-encoded snapshot.json to eliminate redundant storage
- Dominant language
- TypeScript
- Stars
- 35.8k
- Forks
- 1.6k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 4
Description
## Problem
Each migration's `snapshot.json` is a full dump of the entire database schema at that point in time. As a schema grows, each snapshot grows with it — in our project, the latest snapshot is **11,164 lines**. With 105 migrations, that's **618k lines and ~14 MB** of nearly identical JSON files in the repo.
Each snapshot is ~99% identical to the previous one — a single `ALTER TABLE ADD COLUMN` produces an 11k-line file that differs by a handful of lines from its predecessor.
This is distinct from the migration squashing discussion (#3492) and the snapshot-less `--from-db` proposal (#5528). The snapshot chain is the right architecture — it's self-contained, reproducible, and doesn't depend on a live database. The problem is purely the **storage format**.
## Proposal
Store each snapshot as a **delta from the previous snapshot** rather than a full copy. The first snapshot remains complete; subsequent snapshots store only what changed. The full schema at any point can be reconstructed by replaying deltas forward from the first snapshot — the same pattern as git packfiles or incremental backups.
### Concrete options (not mutually exclusive)
1. **JSON delta format**: Store only added/removed/changed keys relative to the previous snapshot. `drizzle-kit` already knows the previous snapshot — it diffs against it to generate SQL.
2. **Compressed snapshots**: gzip or brotli the JSON. Highly repetitive structured data compresses extremely well. Trade human-readability (nobody reads these anyway) for ~95% size reduction.
3. **Single rolling snapshot + journal**: Keep only the latest `snapshot.json` and rely on the migration SQL chain as the history. The journal already tracks ordering.
## Impact
For any project with 50+ migrations and a non-trivial schema, snapshot files dominate the migration folder size. They inflate git repos, slow clones, pollute diffs, and waste tokens when AI coding assistants explore the codebase. The actual migration SQL files — the useful part — are a tiny fraction of the folder.
Contributor guide
Assessment
This issue has not been assessed yet.