benbjohnson / benbjohnson/litestream
Retention never converges on write-idle databases (snapshot dedup + minSnapshotTXID=0 interaction)
- Dominant language
- Go
- Stars
- 14.4k
- Forks
- 417
- Avg merge
- 7d 1h
- Merged PRs (30d)
- 21
Description
## Problem
Two individually-correct changes interact so that retention never converges on write-idle databases:
- #1292 moved snapshot dedup into `Store.CompactDB`: an idle database stops writing new snapshots, so it converges to a **single** retained snapshot whose `CreatedAt` no longer refreshes.
- #1325 made `EnforceSnapshotRetention` return `minSnapshotTXID = 0` when no snapshot exists strictly before the retention timestamp (the safe boundary behavior).
With one aging snapshot, every retention pass yields `minSnapshotTXID = 0`, and `EnforceRetentionByTXID` then retains **all** lower-level files. The database holds its full L1+/L0 file set indefinitely instead of pruning down to the single snapshot. A related edge: when the guard-protected last snapshot is the only survivor of a pass, `minSnapshotTXID` is taken from a snapshot deleted in that same pass, again retaining extra files.
This errs in the safe direction (retains more, never less), but "idle databases never finish pruning" is surprising storage behavior for anyone running many mostly-idle databases.
## Why this needs design discussion rather than a drive-by fix
The obvious fix — when a single snapshot survives, use its `MaxTXID` as the retention floor — is exactly the `else { minSnapshotTXID = info.MaxTXID }` branch that #1325 deliberately dropped from #1312 in favor of returning 0, to avoid pruning files still needed for point-in-time restore before the sole snapshot. Any change must preserve that guarantee.
## Possible directions
- Treat the sole retained snapshot's `MaxTXID` as the floor only once it is older than the full retention window (nothing before it is restorable anyway).
- Have idle databases refresh/re-key the retained snapshot on a slow cadence so retention regains a valid boundary.
- Accept the behavior and document it (storage on idle databases converges to full file set, not one snapshot).
## Testing
`TestDB_EnforceSnapshotRetention_ReturnsZeroWithoutPriorSnapshot` (#1325) pins the current boundary behavior; a fix needs an idle-database convergence test alongside it.
Found during the 2026-07-01 review of the #1290–#1322 stability chain (interaction between #1292 and #1325).
Contributor guide
Research direction
Start with EnforceSnapshotRetention and EnforceRetentionByTXID, then read TestDB_EnforceSnapshotRetention_ReturnsZeroWithoutPriorSnapshot to understand the protected boundary. Agree on a retention-floor design that preserves point-in-time restore safety, and add an idle-database convergence test showing that lower-level files eventually prune appropriately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sqlite
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100