benbjohnson / benbjohnson/litestream

Boundary snapshot encodes entire database under the SQLite write lock, stalling writers on large databases

Open
#1,332 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
14.4k
Forks
417
Avg merge
7d 1h
Merged PRs (30d)
21

Description

Problem

When a checkpoint restarts the WAL and the boundary snapshot cannot be avoided (TRUNCATE mode, and RESTART via CRC64), checkpointWithExecutor re-encodes the entire database into a full L0 LTX while holding the SQLite write lock (the promoted _litestream_lock transaction). All application writers are stalled for the duration of the local encode.

For small databases this is milliseconds. For large databases (10 GB+) it is a multi-second to multi-minute write stall.

#1292 mitigates frequency substantially — the truncate path now tries PASSIVE first and skips TRUNCATE entirely when the passive checkpoint restarts the WAL — but the fall-through still happens under exactly the workload large deployments have: long-lived readers that block a passive restart, combined with a WAL past the truncate threshold.

Failure scenario

  1. Large database with TruncatePageN configured; long-running read transactions block passive WAL restarts.
  2. WAL crosses the truncate threshold; passive checkpoint cannot restart; TRUNCATE fall-through runs.
  3. Boundary snapshot encodes the full database under the write lock; application writes block for the whole encode.

Possible directions (needs design discussion)

  • Encode the boundary snapshot under the existing chkMu/executor read-consistency machinery instead of the write lock, if a consistent read view can be guaranteed across the restart boundary (the write lock is currently what pins the post-restart salts/state).
  • Release the write lock after capturing the WAL header/salts and read pages via the database file descriptor while chkMu excludes further checkpoints.
  • At minimum: document the stall and its bound (database size / local disk throughput) next to TruncatePageN.

The unconditional boundary snapshot on TRUNCATE is load-bearing for correctness (wal_checkpoint(TRUNCATE) reports zero frames, so safety cannot be proven by frame counts — see #1292); any change here must preserve that.

Testing

Regression: snapshot consistency is covered by TestDB_SnapshotReaderConsistentDuringConcurrentCheckpoints (#1166) and the passive-first flow tests (#1292); a fix should keep those green and add a writer-latency assertion under a large-database boundary snapshot.

Found during the 2026-07-01 review of the #1290–#1322 stability chain.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with checkpointWithExecutor and the TRUNCATE/RESTART paths, then read the chkMu/executor consistency machinery and the write-lock handling. Run TestDB_SnapshotReaderConsistentDuringConcurrentCheckpoints and the passive-first flow tests; any finished design must preserve snapshot consistency while preventing the full boundary encode from stalling writers, with a writer-latency regression test for large databases.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sqlite
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.