GC and PRAGMA wal_checkpoint on a lazy clone walk the whole remote graph, cache nothing, and fail offline
- Dominant language
- C
- Stars
- 268
- Forks
- 18
- Avg merge
- 2h 27m
- Merged PRs (30d)
- 447
Description
Found in a full-repo review at `0ba280f06f`.
On a lazy clone, `dolt_gc()` and `PRAGMA wal_checkpoint` mark reachability by fetching every reachable chunk from the remote. Nothing fetched is cached, so the cost is paid again on every run, and the operations cannot complete at all while the origin is unreachable.
Measured on a 636 KB lazy clone of a 23 MB remote with one row materialized:
```
PRAGMA wal_checkpoint on the lazy clone : 160s
lazy clone size before / after : 636886 / 636453 (nothing cached)
PRAGMA wal_checkpoint on the equivalent
non-lazy 23 MB database : 0s
origin OFFLINE:
SELECT v FROM t WHERE id=1 -> works (local cache)
PRAGMA wal_checkpoint -> Error: chunk source I/O error for bc7fa6b9...
SELECT dolt_gc() -> Error: chunk source I/O error for bc7fa6b9...
```
An ordinary SQLite pragma becoming a full-history network walk is surprising on its own; requiring the origin to be reachable for it is worse, because the whole point of a lazy clone is to work from a partial local copy.
## Cause
`gcMarkReachable` calls `chunkStoreGet()` on every reachable hash (`src/doltlite_gc.c:389`). On a lazy store that falls through to `chunkStoreSourceGet()` (`src/chunk_store.c:846`), which goes to the remote. `csSourcePersistMany()` is then skipped because the graph lock is held (`src/doltlite_chunk_source.c:545`), so the fetched chunks are not kept.
`PRAGMA wal_checkpoint` reaches the same compaction: `shimPagerCheckpoint` (`src/pager_shim.c:370`) → `doltliteGcCompactStore` (`src/doltlite_gc.c:1243`). See also #2493, which covers the pragma being wired to GC at all; this issue is about what that costs on a lazy store.
## Fix
Marking on a lazy store should be bounded by what is present locally: a chunk that is not in the local store cannot be swept from it, so it does not need to be fetched to be marked. That makes the checkpoint local, cheap and offline-safe. If a full-graph walk is genuinely wanted for a lazy store, it should be opt-in and should persist what it fetches rather than discarding it.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with gcMarkReachable in src/doltlite_gc.c and follow chunkStoreGet into src/chunk_store.c and chunk_source_persist logic in src/doltlite_chunk_source.c. Trace the PRAGMA path through shimPagerCheckpoint in src/pager_shim.c to doltliteGcCompactStore, then reproduce the lazy-clone and offline-origin cases. Done means local marking stays bounded to locally present chunks, checkpoint and dolt_gc work offline, and repeated runs do not walk the remote graph.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, sqlite
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100