benbjohnson / benbjohnson/litestream
vfs: hydration steady-state apply fetches one page per request — ~600x request amplification and reader stalls on write-heavy workloads
- Dominant language
- Go
- Stars
- 14.4k
- Forks
- 414
- Avg merge
- 7d 1h
- Merged PRs (30d)
- 21
Description
## Summary
When hydration is enabled, the steady-state polling path applies incoming changes to the local hydration file via `Hydrator.ApplyUpdates`, which issues **one ranged `GetObject` request per changed page** while holding the hydrator mutex. This is efficient for sparse deltas, but on write-heavy databases — where most pages of each incoming L0 file are current winners every poll cycle — it:
1. amplifies object-storage request counts by 2–3 orders of magnitude compared to fetching the LTX file once,
2. serializes all fetches behind network RTT inside the mutex, so the replica cannot keep up with the primary, and
3. blocks reads from the hydrated file for the entire apply cycle (`ReadAt` takes the same mutex).
The whole-file apply path already exists (`Hydrator.Restore` / `CatchUp` → `ApplyLTX`) — it just isn't used in steady state. I prototyped a fix that groups updates by source file and fetches files whole above a threshold; it passes the VFS test suite unmodified and removes both the amplification and the reader stalls. Happy to send a PR.
## Environment
- litestream v0.5.14 (relevant code unchanged on current `main`)
- Replica backend: GCS via the S3-compatible endpoint; the behavior applies equally to any per-request-priced backend (S3, R2, …)
- Client: darwin/arm64 for the measurements below; nothing platform-specific
## Workload
A "latest state" style database: continuous batched UPSERTs rewrite most rows every few seconds (device telemetry), sustaining roughly **390 changed pages/s** (4 KB pages). Primary runs with `sync-interval: 60s`, which under this load splits uploads into roughly one L0 file every few seconds (~10–20 files/min). The replica opens the database through the VFS with `HydrationEnabled` and a persistent hydration path, default 1s poll interval.
## Observed (v0.5.14, per-page apply)
An 11-minute window against a live primary:
| metric | value |
|---|---|
| `GetObject` requests | 13,904 |
| distinct LTX files fetched | 23 (**~600 requests per file**) |
| apply throughput | ~21 fetches/s (serialized on RTT inside `h.mu`) |
| rate required to keep up | ~390 pages/s → replica falls behind indefinitely |
| queries completed by a 1 Hz reader | **2 in 11 minutes** (reads blocked on the hydrator mutex) |
Two problems compound here:
- **Request amplification.** ~600 `GetObject`s per LTX file whose pages the replica is going to apply anyway — the file already contains exactly the changed pages, contiguously.
- **Mutex held across network I/O.** `ApplyUpdates` acquires `h.mu` and performs every fetch inside it. `ReadAt` takes the same mutex, so hydrated reads stall for the whole apply cycle; under sustained churn the hydrator effectively never leaves apply.
On request-priced storage this is also a real cost problem: at this workload the per-page path would need on the order of 10⁹ `GetObject`/month per replica to stay current (~$400/month at typical Class B pricing), versus ~$1–2/month fetching files whole. The per-page pattern also multiplies connection churn, which interacts badly with the socket accumulation described in #1354.
## Why per-page exists (and when it is right)
`ApplyUpdates` receives the poll pipeline's winners map (`map[pgno]ltx.PageIndexElem`) and reuses the read path's `FetchPage` primitive — a natural fit, and genuinely optimal for sparse deltas: a compacted L1/L2 file can contain thousands of pages of which only a handful are current winners, and fetching just those avoids downloading megabytes to use kilobytes.
That assumption inverts when nearly every page of an incoming L0 file is a winner — which is simply the steady state for write-heavy databases. There, the delta *is* the file, and per-page fetching turns one download into hundreds of billable round trips.
## Proposal
Keep the winners-map semantics exactly as they are, and change only the fetch strategy inside `Hydrator.ApplyUpdates`:
1. Group the winners map by source LTX file (`Level`, `MinTXID`, `MaxTXID` are already carried by `ltx.PageIndexElem`).
2. If a file's winner count — or better, its winner-bytes / file-size ratio, to protect large compacted files from over-fetching — exceeds a threshold, fetch the file once (`OpenLTXFile(…, 0, 0)`), stream-decode it, and apply **only the winning pages**. Below the threshold, keep today's per-page fetches.
3. Perform all network I/O outside `h.mu`; buffer decoded pages in bounded chunks and take the mutex only for the local `WriteAt`s.
Because each page number has exactly one winner, the per-file page sets are disjoint — apply order across files is irrelevant, and the resulting file bytes are **identical** to the current implementation. Sparse workloads keep exactly today's behavior and request profile.
A possible follow-up (separate change): skip pages whose already-applied TXID is at or beyond the incoming one. Since each level tracks its own position, compaction outputs (L0 → L1 → L2) currently re-deliver data the replica has already applied; TXID-aware skipping would eliminate that redundant transfer as well.
## Prototype results
I implemented the above on current `main` (count threshold of 16 pages, 1,024-page write chunks):
- `go test -tags=vfs ./cmd/litestream-vfs` — hydration e2e, chaos, soak, stress, fuzz, time-travel, and write-integration tests all **pass unmodified**.
- Same workload, 4-minute window: **326 `GetObject` total (~3.6 per file, including page-index reads)**; the replica stays current with the primary in real time; a 0.2 Hz reader completed 49/49 queries with no observable stalls.
| | per-page (v0.5.14) | grouped whole-file (prototype) |
|---|---|---|
| requests per LTX file | ~600 | ~3.6 |
| keeps up with ~390 pages/s churn | no | yes |
| reader stalls during apply | minutes | none observed |
I'd be glad to clean this up into a PR (including making the threshold ratio-based rather than count-based). One incidental note: the root-package `vfs_test.go` currently doesn't compile under `-tags vfs` (`mockReplicaClient` is missing the `SetLogger` method) — happy to fix that alongside if useful.
Contributor guide
Research direction
Start at Hydrator.ApplyUpdates and compare its per-page FetchPage path with Hydrator.Restore/CatchUp and ApplyLTX, which already provide whole-file application. Run go test -tags=vfs ./cmd/litestream-vfs; done means preserving winners-map semantics and passing the hydration, chaos, soak, stress, fuzz, time-travel, and write-integration coverage while avoiding the reported request amplification and reader stalls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sqlite
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100