benbjohnson / benbjohnson/litestream

compaction: single-input compactions duplicate byte-identical LTX files across levels

Open
#1,407 1 comment 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Go
Stars
14.4k
Forks
417
Avg merge
7d 1h
Merged PRs (30d)
21

Description

## Summary

When a compaction has exactly one already-compacted source file, the output is **byte-identical** to the input. The same payload is then stored again at every subsequent level, multiplying storage for no added value.

Raised in #1400:

> so it appears that files which cover the same txid range, are byte-identical by design if the merged ltx files did not overlap in pages mutation. [...] So, it's wasteful of storage to save these as distinct blobs.

## Reproduction

9.7MB database, default configuration, ~3 minutes of light writes:

```
$ md5 rep/ltx/{1,2,3}/0000000000000001-0000000000000001.ltx
MD5 (rep/ltx/1/0000000000000001-0000000000000001.ltx) = bbf1b0721b05d657fb00ecd65f5a459e
MD5 (rep/ltx/2/0000000000000001-0000000000000001.ltx) = bbf1b0721b05d657fb00ecd65f5a459e
MD5 (rep/ltx/3/0000000000000001-0000000000000001.ltx) = bbf1b0721b05d657fb00ecd65f5a459e
```

Three identical 8.4MB copies. Per-level disk usage:

```
8.1M rep/ltx/0
8.0M rep/ltx/1
8.0M rep/ltx/2
8.0M rep/ltx/9
```

~32MB of replica storage for a 9.7MB database.

## Scope of the identity

This holds only for **already-canonical** inputs, i.e. L1 → L2 → L3. It does **not** hold for L0 → L1:

```
MD5 (rep/ltx/0/0000000000000010-0000000000000010.ltx) = 0bd4ab43891745c9a3ba9de3ee9a41e3
MD5 (rep/ltx/1/0000000000000010-0000000000000010.ltx) = 8f725caeae7cd03312031ed38e17ab90
```

L0 files carry WAL-origin metadata (`WALOffset`, `WALSize`, `WALSalt1/2`, `NodeID`) that `ltx.Compactor` deliberately drops, so the re-encode differs. Snapshots (L9) use a separate full-database encoder and are also not identical.

## Cause

`Compactor.Compact()` selects every source-level file above the destination's current max TXID and hands them all to `ltx.NewCompactor`. When exactly one qualifies, the result is a pure re-serialization.

Note the reporter's proposed cause — a size threshold making large files ineligible for merging — is **not** what is happening. There is no size threshold anywhere in this path. The real trigger is timing: at the moment L2 ran, only one L1 file existed above its watermark.

## Possible directions

- Skip promotion until more than one source file is available at the destination's watermark.
- Store a pointer/alias at the higher level instead of a second copy.
- Use a server-side copy where the backend supports it, to at least avoid re-upload cost.

Each has consequences for restore planning and for level-wise retention (which currently assumes every level is independently walkable), so this needs a design call rather than a direct patch.

Filing as an optimization proposal, not a correctness bug.

Not the same problem as #1172 (that is about reducing S3 LIST operations via manifests).

Related: #1400

Contributor guide

Open the contributing guide

Research direction

Start with Compactor.Compact and ltx.NewCompactor, then trace how level-wise retention and restore planning use independently walkable levels. The issue needs a design decision among deferring promotion, aliasing, or server-side copy; it is done when the chosen approach avoids redundant storage without breaking those assumptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, sqlite
Domain
cloud, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.