dolthub / dolthub/doltlite

Add per-chunk compression (snappy) for storage size at GB scale

Open
#655 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
268
Forks
18
Avg merge
2h 26m
Merged PRs (30d)
454

Description

## Summary

DoltLite stores chunks uncompressed. Dolt's NBS wraps each chunk record as `length + snappy(data) + crc32` (`go/store/nbs/table.go`); we don't. For typical tabular data this is a ~50% storage delta; for repetitive payloads (JSON, logs, long-string schemas) closer to 70–80%.

Don't ship this preemptively. File it as a marker for when the first real storage-size complaint at GB scale lands.

## Why we skipped it originally

1. **Simplicity / no deps.** Vendoring a compression library means every embed target (iOS, Android, WASM, the language bindings in #627–#632) has to deal with it. Single-file, no-deps SQLite-style is the brand.
2. **Scale ceiling matches the workload.** DoltLite targets embedded / local-first / browser. Realistic DBs are single-digit GB. Saving 5 GB on a 10 GB DB matters at server scale; saving 200 MB on a 400 MB DB is a rounding error at laptop scale.
3. **WASM cost.** Decompression on every chunk read is a hot-path tax. WASM JITs aren't great at tight decompression loops, and OPFS reads aren't free either.

## When to revisit

- A user hits an actual storage-size complaint at multi-GB scale, OR
- We start shipping DoltLite for use cases (replicas, archival snapshots) where storage cost dominates read latency

## Implementation sketch (when the time comes)

The chunk record format is already `length_le32(4) + data(length)` (`src/chunk_store.h:11`). Steal a high bit of the length word for a "compressed" flag, slot snappy in the data section. Native build vendors snappy (~1500 LoC, BSD-3); WASM build can disable via a compile-time flag if the decompression cost flips the cost-benefit.

- On-disk format bump: `CHUNK_STORE_VERSION` 10 → 11.
- Snappy encoder/decoder vendored at `src/snappy.{h,c}` or `ext/snappy/`.
- Build wiring: `main.mk` adds the new objects to `PROLLY_OBJS`; WASM build can pass `-DDOLTLITE_NO_COMPRESSION` to elide the dependency.
- Compatibility: 0.9.x DBs (uncompressed) won't open in the bumped version unless we keep a fallback decoder for `compressed flag = 0`. Worth doing — keeps existing DBs readable, just doesn't recompress them on-the-fly.

## Tradeoffs to measure before merging

- **Read perf delta on warm cache:** decompression cost vs. fewer bytes paged in. Likely net loss for fully-resident workloads.
- **WASM read perf delta:** measure separately. May warrant the disable flag.
- **Write perf delta:** snappy adds a pass on every chunk emission. Probably 5–15% slower commits.
- **Storage savings on TPC-C / sysbench / a real DoltHub schema dump:** the actual win we'd be selling.

## Alternatives to keep in mind

- **Dictionary-based zstd** (closer to Dolt's archive format) — better ratios, more code, slower decode. Probably the right move for cold archival paths only, not the live chunk store.
- **Page-level compression at the OS layer** (filesystem compression on macOS APFS, Btrfs, ZFS) — free for users on those filesystems, useless on others. Not portable, but worth mentioning to anyone hitting size pressure.

## Acceptance criteria

- Snappy encoder + decoder vendored under a permissive license, no runtime deps.
- Format-version bump with clear error on opening older DBs.
- Compile-time flag to disable for WASM (or anywhere else cost-benefit flips).
- Storage size on a representative DB drops by at least 40%.
- Read perf within 1.2× of master; write perf within 1.2× of master.

Contributor guide

No contributing guide indexed for this repository

Research direction

Revisit this only after a multi-GB storage complaint or a storage-dominated use case appears. Start with the chunk record format in src/chunk_store.h and build wiring in main.mk, then measure read, write, WASM, and storage-size tradeoffs against the stated acceptance criteria before choosing the format and compatibility approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.