drizzle-team / drizzle-team/drizzle-orm

Feature: delta-encoded snapshot.json to eliminate redundant storage

Open
#5,635 4 comments 21 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
35.8k
Forks
1.6k
Avg merge
2d 7h
Merged PRs (30d)
4

Description

## Problem

Each migration's `snapshot.json` is a full dump of the entire database schema at that point in time. As a schema grows, each snapshot grows with it — in our project, the latest snapshot is **11,164 lines**. With 105 migrations, that's **618k lines and ~14 MB** of nearly identical JSON files in the repo.

Each snapshot is ~99% identical to the previous one — a single `ALTER TABLE ADD COLUMN` produces an 11k-line file that differs by a handful of lines from its predecessor.

This is distinct from the migration squashing discussion (#3492) and the snapshot-less `--from-db` proposal (#5528). The snapshot chain is the right architecture — it's self-contained, reproducible, and doesn't depend on a live database. The problem is purely the **storage format**.

## Proposal

Store each snapshot as a **delta from the previous snapshot** rather than a full copy. The first snapshot remains complete; subsequent snapshots store only what changed. The full schema at any point can be reconstructed by replaying deltas forward from the first snapshot — the same pattern as git packfiles or incremental backups.

### Concrete options (not mutually exclusive)

1. **JSON delta format**: Store only added/removed/changed keys relative to the previous snapshot. `drizzle-kit` already knows the previous snapshot — it diffs against it to generate SQL.
2. **Compressed snapshots**: gzip or brotli the JSON. Highly repetitive structured data compresses extremely well. Trade human-readability (nobody reads these anyway) for ~95% size reduction.
3. **Single rolling snapshot + journal**: Keep only the latest `snapshot.json` and rely on the migration SQL chain as the history. The journal already tracks ordering.

## Impact

For any project with 50+ migrations and a non-trivial schema, snapshot files dominate the migration folder size. They inflate git repos, slow clones, pollute diffs, and waste tokens when AI coding assistants explore the codebase. The actual migration SQL files — the useful part — are a tiny fraction of the folder.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.