Same-commit re-crawl is unusably slow on large repositories

Re-crawling a finished database at the same commit should be close to free: nothing has changed, so the crawl should verify that and stop. @davidh233's team measured 67 to 92s on a small repository, and on a larger one the classify stage alone had not finished after 19 minutes. That makes incremental refresh impractical at monorepo scale, which is most of the reason to run a crawl on a schedule at all.

The cost is in the read path, and it is the mirror image of #83. There is no scalar index on row_id, so every point lookup scans every fragment of the whole table. The crawl does two of those per file: classify looks up each file's sentinel to decide skip versus redo, and add_label reads back each existing file's chunk rows to confirm they still carry the label. Neither stage writes anything on a same-commit re-crawl. They read everything to find there is nothing to do.

Fragment count multiplies all of it, since a scan crosses every fragment, and fragments accumulate with every checkpoint commit.

Investigation

Their measurements, per-file against batched, on two copies of the same database:

Batching is worth two orders of magnitude on the scan count. Compaction is worth another large factor on top, which is a separate problem and gets its own issue.

Proposed fix

Both stages get the same treatment as #83: replace per-file lookups with row_id IN (...) and file_id IN (...) predicates over bounded batches.

Four pieces, one PR:

1. Batched sentinel read. A get_sentinel_statuses helper taking a slice of sentinel row_ids and returning a status map. Classify computes all file_ids locally, fetches the map in one batched call, then classifies in the original file order so new_files still comes out in upstream order.

2. Batched chunk read for add_label. A get_chunks_by_file_ids helper batching by file_id IN, regrouping rows per file before the existing per-file verification runs. This is the piece with real logic in it.

3. Batch size. Reuse the existing UPSERT_BATCH_SIZE rather than introducing 200 as a second constant, unless there is a measured reason it differs here.

4. No per-file fallback on batch failure. Their version catches a batch-level error and retries that batch row by row. I would rather not: a failed read is a real error, retrying per file turns it into a slow success nobody notices, and it doubles the paths through classify while being almost never exercised. @davidh233, if you hit something concrete that motivated the fallback, I would like to know what it was, since if it was predicate length that is a batch-size question with a different answer.

Related

#83 batches the write side of the same pattern, the per-file completion marks at checkpoints.

#84 retires the sentinel mechanism, which removes piece 1 entirely. Piece 2 survives it, since the label verification read has nothing to do with the sentinel. This issue is worth doing first regardless: #84 is a schema change with a longer investigation ahead of it, and re-crawls are unusable on large repositories until then.

@LPegasus

Start at the classify and add_label read paths, then inspect the existing UPSERT_BATCH_SIZE usage and the batching approach from issue #83. Implement the batched sentinel and chunk reads while preserving file order and existing per-file verification; done means same-commit re-crawls avoid per-file scans and retain their current behavior.