benbjohnson / benbjohnson/litestream

Retention never converges on write-idle databases (snapshot dedup + minSnapshotTXID=0 interaction)

Open
#1,333 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14.4k
Forks
417
Avg merge
7d 1h
Merged PRs (30d)
21

Description

## Problem

Two individually-correct changes interact so that retention never converges on write-idle databases:

- #1292 moved snapshot dedup into `Store.CompactDB`: an idle database stops writing new snapshots, so it converges to a **single** retained snapshot whose `CreatedAt` no longer refreshes.
- #1325 made `EnforceSnapshotRetention` return `minSnapshotTXID = 0` when no snapshot exists strictly before the retention timestamp (the safe boundary behavior).

With one aging snapshot, every retention pass yields `minSnapshotTXID = 0`, and `EnforceRetentionByTXID` then retains **all** lower-level files. The database holds its full L1+/L0 file set indefinitely instead of pruning down to the single snapshot. A related edge: when the guard-protected last snapshot is the only survivor of a pass, `minSnapshotTXID` is taken from a snapshot deleted in that same pass, again retaining extra files.

This errs in the safe direction (retains more, never less), but "idle databases never finish pruning" is surprising storage behavior for anyone running many mostly-idle databases.

## Why this needs design discussion rather than a drive-by fix

The obvious fix — when a single snapshot survives, use its `MaxTXID` as the retention floor — is exactly the `else { minSnapshotTXID = info.MaxTXID }` branch that #1325 deliberately dropped from #1312 in favor of returning 0, to avoid pruning files still needed for point-in-time restore before the sole snapshot. Any change must preserve that guarantee.

## Possible directions

- Treat the sole retained snapshot's `MaxTXID` as the floor only once it is older than the full retention window (nothing before it is restorable anyway).
- Have idle databases refresh/re-key the retained snapshot on a slow cadence so retention regains a valid boundary.
- Accept the behavior and document it (storage on idle databases converges to full file set, not one snapshot).

## Testing

`TestDB_EnforceSnapshotRetention_ReturnsZeroWithoutPriorSnapshot` (#1325) pins the current boundary behavior; a fix needs an idle-database convergence test alongside it.

Found during the 2026-07-01 review of the #1290–#1322 stability chain (interaction between #1292 and #1325).

Contributor guide

Open the contributing guide

Research direction

Start with EnforceSnapshotRetention and EnforceRetentionByTXID, then read TestDB_EnforceSnapshotRetention_ReturnsZeroWithoutPriorSnapshot to understand the protected boundary. Agree on a retention-floor design that preserves point-in-time restore safety, and add an idle-database convergence test showing that lower-level files eventually prune appropriately.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sqlite
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.