dolthub / dolthub/doltlite

GC and PRAGMA wal_checkpoint on a lazy clone walk the whole remote graph, cache nothing, and fail offline

Closed
#2,905 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
268
Forks
18
Avg merge
2h 27m
Merged PRs (30d)
447

Description

Found in a full-repo review at `0ba280f06f`.

On a lazy clone, `dolt_gc()` and `PRAGMA wal_checkpoint` mark reachability by fetching every reachable chunk from the remote. Nothing fetched is cached, so the cost is paid again on every run, and the operations cannot complete at all while the origin is unreachable.

Measured on a 636 KB lazy clone of a 23 MB remote with one row materialized:

```
PRAGMA wal_checkpoint on the lazy clone : 160s
lazy clone size before / after : 636886 / 636453 (nothing cached)
PRAGMA wal_checkpoint on the equivalent
non-lazy 23 MB database : 0s

origin OFFLINE:
SELECT v FROM t WHERE id=1 -> works (local cache)
PRAGMA wal_checkpoint -> Error: chunk source I/O error for bc7fa6b9...
SELECT dolt_gc() -> Error: chunk source I/O error for bc7fa6b9...
```

An ordinary SQLite pragma becoming a full-history network walk is surprising on its own; requiring the origin to be reachable for it is worse, because the whole point of a lazy clone is to work from a partial local copy.

## Cause

`gcMarkReachable` calls `chunkStoreGet()` on every reachable hash (`src/doltlite_gc.c:389`). On a lazy store that falls through to `chunkStoreSourceGet()` (`src/chunk_store.c:846`), which goes to the remote. `csSourcePersistMany()` is then skipped because the graph lock is held (`src/doltlite_chunk_source.c:545`), so the fetched chunks are not kept.

`PRAGMA wal_checkpoint` reaches the same compaction: `shimPagerCheckpoint` (`src/pager_shim.c:370`) → `doltliteGcCompactStore` (`src/doltlite_gc.c:1243`). See also #2493, which covers the pragma being wired to GC at all; this issue is about what that costs on a lazy store.

## Fix

Marking on a lazy store should be bounded by what is present locally: a chunk that is not in the local store cannot be swept from it, so it does not need to be fetched to be marked. That makes the checkpoint local, cheap and offline-safe. If a full-graph walk is genuinely wanted for a lazy store, it should be opt-in and should persist what it fetches rather than discarding it.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with gcMarkReachable in src/doltlite_gc.c and follow chunkStoreGet into src/chunk_store.c and chunk_source_persist logic in src/doltlite_chunk_source.c. Trace the PRAGMA path through shimPagerCheckpoint in src/pager_shim.c to doltliteGcCompactStore, then reproduce the lazy-clone and offline-origin cases. Done means local marking stays bounded to locally present chunks, checkpoint and dolt_gc work offline, and repeated runs do not walk the remote graph.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, sqlite
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.