benbjohnson / benbjohnson/litestream
compaction: single-input compactions duplicate byte-identical LTX files across levels
- Dominant language
- Go
- Stars
- 14.4k
- Forks
- 417
- Avg merge
- 7d 1h
- Merged PRs (30d)
- 21
Description
## Summary
When a compaction has exactly one already-compacted source file, the output is **byte-identical** to the input. The same payload is then stored again at every subsequent level, multiplying storage for no added value.
Raised in #1400:
> so it appears that files which cover the same txid range, are byte-identical by design if the merged ltx files did not overlap in pages mutation. [...] So, it's wasteful of storage to save these as distinct blobs.
## Reproduction
9.7MB database, default configuration, ~3 minutes of light writes:
```
$ md5 rep/ltx/{1,2,3}/0000000000000001-0000000000000001.ltx
MD5 (rep/ltx/1/0000000000000001-0000000000000001.ltx) = bbf1b0721b05d657fb00ecd65f5a459e
MD5 (rep/ltx/2/0000000000000001-0000000000000001.ltx) = bbf1b0721b05d657fb00ecd65f5a459e
MD5 (rep/ltx/3/0000000000000001-0000000000000001.ltx) = bbf1b0721b05d657fb00ecd65f5a459e
```
Three identical 8.4MB copies. Per-level disk usage:
```
8.1M rep/ltx/0
8.0M rep/ltx/1
8.0M rep/ltx/2
8.0M rep/ltx/9
```
~32MB of replica storage for a 9.7MB database.
## Scope of the identity
This holds only for **already-canonical** inputs, i.e. L1 → L2 → L3. It does **not** hold for L0 → L1:
```
MD5 (rep/ltx/0/0000000000000010-0000000000000010.ltx) = 0bd4ab43891745c9a3ba9de3ee9a41e3
MD5 (rep/ltx/1/0000000000000010-0000000000000010.ltx) = 8f725caeae7cd03312031ed38e17ab90
```
L0 files carry WAL-origin metadata (`WALOffset`, `WALSize`, `WALSalt1/2`, `NodeID`) that `ltx.Compactor` deliberately drops, so the re-encode differs. Snapshots (L9) use a separate full-database encoder and are also not identical.
## Cause
`Compactor.Compact()` selects every source-level file above the destination's current max TXID and hands them all to `ltx.NewCompactor`. When exactly one qualifies, the result is a pure re-serialization.
Note the reporter's proposed cause — a size threshold making large files ineligible for merging — is **not** what is happening. There is no size threshold anywhere in this path. The real trigger is timing: at the moment L2 ran, only one L1 file existed above its watermark.
## Possible directions
- Skip promotion until more than one source file is available at the destination's watermark.
- Store a pointer/alias at the higher level instead of a second copy.
- Use a server-side copy where the backend supports it, to at least avoid re-upload cost.
Each has consequences for restore planning and for level-wise retention (which currently assumes every level is independently walkable), so this needs a design call rather than a direct patch.
Filing as an optimization proposal, not a correctness bug.
Not the same problem as #1172 (that is about reducing S3 LIST operations via manifests).
Related: #1400
Contributor guide
Research direction
Start with Compactor.Compact and ltx.NewCompactor, then trace how level-wise retention and restore planning use independently walkable levels. The issue needs a design decision among deferring promotion, aliasing, or server-side copy; it is done when the chosen approach avoids redundant storage without breaking those assumptions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, sqlite
- Domain
- cloud, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100