Crawl runs out of memory and accumulates thousands of table versions during checkpoints

@davidh233's team hit this indexing a 48k-file repository: the crawl was OOM-killed at 74% progress with anon-rss at 32.4 GB against a 31 GiB ceiling. Resuming did not help, because the resumed crawl had to run 48k per-file lookups against a chunks table that had accumulated 11,856 versions, and showed no progress after 2.4 hours.

The cause is in upload_and_mark_complete in src/app/crawl/pipeline.rs. Every 60 seconds the checkpoint flushes finished work, then marks each completed file by calling update_file_complete once per file. In LanceDB each of those is a full-table predicate scan plus its own commit, roughly 91 of each per checkpoint. Two things follow. The scan's transient memory grows with the table and holds at its high-water mark, so RSS climbs at every checkpoint boundary and never comes down. And the version count grows by about 91 per checkpoint, which slows every subsequent read, including the per-file lookups a resumed crawl depends on.

Investigation

Using a synthetic sample of 16k same-size chunks to remove sequence length as a variable, all growth lands on checkpoint boundaries and each increment gets larger as the table grows: +23 MB at checkpoint 1, +80 MB at checkpoint 16. The database reached 1,219 versions after 13 checkpoints.

They also tested whether merge_insert was contributing. With batched marking in place and writes forced back through merge_insert, all 25 checkpoints stayed within 4 MB per increment. So the write path is not the driver; the per-item update commits are.

After batching, on the same synthetic sample: per-checkpoint RSS increment goes from +23 to +80 MB growing with the table, to +1 to +2 MB constant. Versions over 10 checkpoints go from about 900 to 21. Throughput improves slightly, 1.4 to 1.67 chunks/s. At full scale their 186k-chunk crawl then completed in 11h12m with no OOM and no retries, ending at 1,343 versions instead of 11,856 at 74%. Database size dropped as a side effect, 374 MB to 60 MB on a smaller repository, because version history no longer pins dead fragments.

Proposed fix

Add a batched update_files_complete to src/engine/storage/chunks/storage.rs taking a row_id IN (...) predicate, and have the checkpoint teardown collect completed files and mark them in one call. Their implementation:

pub async fn update_files_complete(&self, row_ids: &[String], complete: bool) -> Result<()> {
    if row_ids.is_empty() {
        return Ok(());
    }
    let _commit_guard = acquire_commit_mutex(&self.db_path)?;
    let value = if complete { "true" } else { "false" };
    for batch in row_ids.chunks(200) {
        let refs: Vec<&str> = batch.iter().map(|s| s.as_str()).collect();
        let predicate = in_quoted_strs("row_id", &refs);
        self.table
            .update()
            .only_if(&predicate)
            .column("file_complete", value)
            .execute()
            .await
            .map_err(|e| anyhow!("Failed to update file_complete (batch): {}", e))?;
    }
    Ok(())
}

Two changes I would make to this: The commit mutex should be acquired inside the batch loop rather than around the whole loop: the discipline elsewhere in the storage layer is one acquisition per LanceDB write, and holding it across every batch blocks writers against other catalogs for the duration. And the batch size should reuse the existing UPSERT_BATCH_SIZE rather than introduce a second batching constant, unless there is a measured reason 200 beats 1000 here.

A scalar BTree index on row_id would be the other way to fix this, since it removes the full-table scan that makes per-file updates expensive in the first place. I am not proposing it: it adds index maintenance to every write path, its interaction with merge_insert and with compaction is uncharacterized here, and the scan cost is dominated by fragment count anyway. Batching is the smaller change with no new surface.

Side note: the completion marker only exists because chunk 1 of each file carries a file_complete flag, which is a workaround from an earlier storage backend that had no cross-row atomicity. Removing that mechanism deletes this write pass rather than batching it, and I have opened a separate issue for it.

@LPegasus

Start in src/app/crawl/pipeline.rs at upload_and_mark_complete, then read src/engine/storage/chunks/storage.rs and the existing UPSERT_BATCH_SIZE and commit-mutex usage. Trace how completed files are collected and marked, and verify that batching reduces per-checkpoint commits and version growth without changing completion behavior.