Re-crawl cost scales with repository size instead of with what changed

Re-crawling a label does the same amount of database work whether one file changed or ten thousand did. Classify computes a file_id for every file in the tree and looks up its sentinel row; add_label reads back the chunk rows of every file that already exists to confirm they still carry the label. Git already knows which paths changed. Monodex asks the database anyway, once per file, every round.

@davidh233's team measured what that costs. On a 230k-file repository, a re-crawl at the same commit with nothing changed takes 81 to 93s: about 26s to compute identities and verify sentinels, and 50 to 64s to read back label membership. Nothing is written. On a 48k-file repository the same floor is 15s. It scales with repository size rather than with the size of the change, so on their setup a one-file edit costs 94s, of which about one second is the work.

That floor also forces a policy on their local watcher. Refreshes have to be batched until enough has changed to amortize 93s, which is the opposite of what a developer wants from an index meant to track their working tree.

Their patch

They added what they called a crawl cursor, with no new storage. <method>_source records the commit of the round and <method>_complete flips true only after every stage succeeds, so the pair already says "this commit was fully indexed." On the next crawl they enumerate that commit's tree with gix, measured at 0.3s on 230k files, and compare it to the current tree by path and blob:

pub fn is_unchanged(&self, file: &FileEntry) -> bool {
    self.blob_by_path.get(&file.relative_path) == Some(&file.blob_id)
}

A file that matches skips the sentinel lookup and, as a consequence, the label read-back: verification only covers the existing set, and skipped files are not in it. Their patch also handles the trap that creates, which is that a skipped file still has to be counted into the touched set:

// Files skipped by the cursor must count into the touched set,
// otherwise label cleanup would treat them as stale and strip their label.
let mut existing_and_skipped: HashSet<String> =
    classify_output.existing_file_ids.clone();
existing_and_skipped.extend(
    classify_output.cursor_skipped_file_ids.iter().cloned());

Toggling it on their fork, which already carries their batching fixes, so the left column is faster than today's main:

Scenario	Off	On
Same-commit re-crawl, 230k files	81s	6s
Local edit, 10 files	93s	12s
Real catch-up, 230 files over 55 hours	372s	297s

Equivalence held in all three: same chunk set, same reuse hits, same FTS additions and removals, same retrieval results.

The direction is right. A crawl should apply what changed to an already-indexed label instead of re-deriving the label's whole state and comparing. The part I want to change is where the previous state comes from.

Why deriving the previous state from Git does not hold up

Their correctness argument is that file_id incorporates blob_id, so a file Git reports as unchanged has an unchanged file_id and last round's verified state still applies. That is true of blob_id and not of file_id. classify_files in src/app/crawl/phases.rs computes it from EMBEDDER_ID and CHUNKER_ID as well, which are the constants the project uses to force re-indexing. Four cases where the same path and blob does not mean the stored rows are usable:

• A chunker or embedder bump. Every file_id is new and none of the old rows match, but complete=true from the last round survives on disk and the tree is unchanged, so every file is skipped. Worse than stale results: the skipped files enter the touched set under their new file_ids, so label reassignment finds the old rows untouched and strips the label from all of them. The crawl reports success and empties the label. #82 bumps EMBEDDER_ID, so this case will arrive.
• Widening the retrieval selection. An FTS-only crawl writes rows with NULL vectors, which is why the sentinel check consults has_vector when vector is selected. Comparing trees bypasses that check, so a crawl adding vector to an FTS-only label would skip every unchanged file and produce a label whose vectors never arrive.
• Files that failed last round. chunk_new_files turns read failures, non-UTF-8 content, and chunking errors into warnings and continues, while <method>_complete comes from phase success and label reassignment, not from those warnings. Today the file has no sentinel, so the next crawl retries it and warns again. Under a tree comparison it is unchanged, so it is skipped and the warning never returns. For a deterministically bad file the retry was never going to succeed, so what is lost is the warning, which is the part that matters.
• A crawl-config change. patternsToExclude, patternsToKeep, and fileTypes decide which files are eligible and change with no commit. A newly included file has the path and blob it always had and has never been indexed.

The common thread is that <method>_complete means "this retrieval method's phase finished," and this needs it to mean "every eligible file is materialized under the current rules." Those coincide only while nothing changes the rules between rounds, and each case above is a way the rules change.

Record what was indexed instead of inferring it

Rush solves the same problem and does not diff commits to do it. getDetailedRepoStateAsync in @rushstack/package-deps-hash builds the current state of the repository as a map from path to blob hash, using git ls-files --cached for the index (no file reads), git status -u --no-renames for what is dirty, and git hash-object --stdin-paths for only those. InputsSnapshot then folds a project's file hashes together with its environment variables, Node version, output folder names, and rush-project.json into one hash, and compares that against the value recorded for the last build. No commit appears anywhere in the mechanism. Committed and uncommitted states are the same case, and tool-side inputs invalidate through the same path as file contents rather than through separate rules.

Applied here, the proposal is that a successful crawl writes down what it indexed, and the next crawl compares against that record rather than reconstructing it from Git.

The crawl inventory. One per label, holding a header and a body. The header records the identity constants (EMBEDDER_ID, CHUNKER_ID), a hash of the compiled crawl config, which retrieval methods were materialized, and the source it was taken from. The body is the sorted list of every file that was successfully indexed, as path and blob ID. On a 230k-file repository that is roughly 25MB, written once at finalization and read as one sequential file.

Written only on full success, atomically by temp file and rename, following the monodex-meta.json and FTS manifest.json precedent for tool-managed sidecar state. purge removes it with the rest of the catalog's state.

The next crawl. If there is no inventory, or its identity constants or config hash differ from the current ones, or the current retrieval selection includes a method the inventory does not cover, the crawl runs exactly today's full verification and writes a fresh inventory at the end. Otherwise it computes the current path-to-blob map and compares:

• Same path, same blob: skip the sentinel lookup and the label read-back, and count the file_id into the touched set.
• Different blob, or absent from the inventory: process as today.
• Present in the inventory and absent now: a deletion, discussed below.

Where the current map comes from. In commit mode, a gix tree walk, which their patch measures at 0.3s on 230k files. In working-directory mode, the git ls-files and git status and git hash-object sequence in src/engine/git_ops/working_dir.rs, which every working-dir crawl already runs and then discards. So working-directory labels are covered by the same mechanism, at no additional enumeration cost, which matters because the watcher is where the 93s floor is felt most.

Three of the four problems above disappear rather than being guarded against. Identity constants and config are in the header, so a bump invalidates the inventory wholesale. Method coverage is in the header, so widening the selection falls back for one round. And a file that failed last round was never indexed, so it is not in the inventory, so it comes back as new and warns again. There is no list of preconditions to keep in sync with future changes; the record either describes the current rules or it does not.

One case here predates this issue. fileTypes decides which chunking strategy a file gets, and strategy is not part of file_id, so today changing a file's mapping leaves its old chunks in place on the next crawl. Putting the config hash in the header closes that for the fast path, and the general problem stays open.

Two mechanical points

The inventory has to be read before step 1 of the crawl, since the label upsert overwrites the source fields and clears the completion flags before file processing starts.

Skipped files must count into the touched set, per their patch above. This is the one error in the design that is invisible until a label has quietly lost its content, so it wants a regression test rather than only a comment.

Not in this issue

Deletions. The inventory makes them nearly free: a path present in the inventory and absent now has a known old (path, blob), therefore a known old file_id, so its label membership could be dropped directly instead of being discovered by the label reassignment scan at step 5. That would make the scan a fallback rather than the normal algorithm, and it changes interrupted-crawl semantics and touches the same code as #87 and #88. Next step after this, separately.

The FTS phase. It reconciles against Tantivy's term dictionary independently and is measured in single-digit seconds. Nothing here changes it.

Sequencing

This adds a sidecar rather than a column, so it does not force a schema bump and does not have to ride with #82, #84, and #90. It is independent of all of them.

#85 still earns its place regardless. Every round the inventory declines to run, and every changed file in the rounds where it does run, goes through the batched reads.

Open questions

One inventory file per label is the obvious unit, but a catalog with many labels then stores the same path list many times over, since most labels share most of their files. A per-catalog file with per-label membership would deduplicate that, at the cost of a more complicated write path. I do not have a strong view yet, and 25MB per label is small enough that I would ship the simple version first.

The format is also open. Sorted plain text is diffable and easy to inspect by hand, and at 230k entries a binary or compressed encoding would be several times smaller. I lean toward the readable one, on the grounds that a tool-managed file nobody can read when something goes wrong costs more than the disk does.

A note on naming: their patch calls this a crawl cursor. I renamed it because a cursor already means something specific in LanceDB, in Tantivy, and in the Git APIs this code calls.

Start by tracing crawl state and file identity handling in src/app/crawl/phases.rs, then inspect the existing working-directory Git enumeration in src/engine/git_ops/working_dir.rs. Compare the proposed inventory lifecycle with the monodex-meta.json and FTS manifest precedents; done means unchanged files avoid redundant verification without changing crawl results, and skipped files remain in the touched set.