Embeddings are recomputed for chunks whose text has not changed

Monodex decides what to re-embed at file granularity. A file's identity folds in its blob, so changing one line gives the file a new file_id, gives every one of its chunks a new row_id, and sends all of them back through the embedder. That is correct, and for a file whose chunks mostly did change it is also the cheapest thing to do. The assumption underneath it is that a changed file is mostly changed content.

@davidh233's team measured how well that assumption holds on a real monorepo, and it holds less often than expected. Embedding is 86 to 89% of incremental crawl time, and much of that turns out to be recomputing vectors for chunk texts that are byte-identical to what is already stored. Across 200 commits and 837 changed files, 15% were pure renames: same bytes, same chunks, every vector recomputed. On a single day's increment of 667 changed files and 3,443 chunks, more than half the chunks had text already present in the database.

Investigation

Their patch builds a lookup at the start of an incremental crawl. It fetches the old chunks for the changed paths, keeps a map from sha256(chunk text) to vector, and has the embedding worker check the map before running inference. The vector is a pure function of the chunk text, so a hit is exact rather than approximate.

Their worker-side check, in src/app/crawl/pipeline.rs. This is the piece the proposal below keeps as-is; the map it consults is built per crawl by get_vectors_by_relative_paths, which fetches old chunks for the changed paths in relative_path IN batches with an embedder filter, keeping only the hash-to-vector pairs and never the text:

     .for_each(|(idx, chunk)| {
         let worker_index = idx % num_workers;
+        if let Some(map) = &vector_reuse {
+            let key = crate::engine::identity::compute_hash(&chunk.text);
+            if let Some(v) = map.get(&key) {
+                let _ = embed_tx.send((chunk, v.clone()));
+                processed_clone.fetch_add(1, Ordering::Relaxed);
+                reused_clone.fetch_add(1, Ordering::Relaxed);
+                return;
+            }
+        }
         match embedder.encode(&chunk.text, worker_index) {

Measured on that one-day increment, on and off against the same machine:

Building the map cost 0.2s. Retrieval results matched on both sides. On macOS against a different commit pair they saw 31% reuse, and on two real wide diffs 74 and 79%.

They flagged one gap themselves: the lookup fetches candidates by relative_path, so the pure renames that motivated the work are exactly the case it misses. A renamed file's old chunks are still in the database under the old path, and the query never looks there.

Proposal

Store the hash rather than deriving it per crawl, and key the lookup on content instead of path.

A new embed_input_hash column on the chunks table. It holds sha256 of the exact string handed to the embedder. Today that string is the chunk text, so the column's contents are what their map computes in memory; storing it means the database can be queried by content directly, without reading text or vectors back to rebuild a map.

Lookup keyed on (EMBEDDER_ID, embed_input_hash), across the whole database. Both halves matter. The embedder component is required because a vector is only a valid answer for the embedder that produced it. Dropping the path scoping is what closes the rename gap: a renamed file's chunks hash the same as before, so they hit, and so do identical chunks that live in a different file entirely.

A grouping pass over the crawl's own chunks. Chunks are all partitioned before embedding starts, so before the parallel loop runs, group the pending chunks by the same key and embed one member of each group. This covers duplicates that are new in this crawl and therefore cannot be in the database yet, which is the common shape on a first crawl of a repository with vendored or generated code.

Fan out the vector only. A reused vector is copied onto each chunk's own row. Everything else on that row is specific to where the chunk lives (path, package, ordinals, breadcrumb, line numbers, label membership) and comes from that chunk, not from the row the vector came from.

Vectors stay on chunk rows rather than moving to a table of their own. Colocation is what makes label-scoped vector search a single scan, since the filter column and the vector sit on the same row, and it keeps collection to one rule: a row with no remaining labels is collectable. A shared vector table would turn every search into a cross-table operation and every collection into a reference count. The duplicate copies cost disk; what this proposal is trying to avoid spending is inference.

Sequencing

This wants to land with #82 and #84. #82 lowers MAX_LENGTH and bumps EMBEDDER_ID, which is not just good practice here: a vector produced under the old truncation cap is a different vector for the same text, so without the bump this lookup would serve stale results with nothing recording which cap produced them. #84 changes the schema for its own reasons. All three force a rebuild, and doing them together means one rebuild rather than three.

The whole-database lookup also wants the scalar index from #89 on the new column, otherwise each lookup is a full-table scan.

Acceptance

Reuse produces bit-identical vectors by definition, so the bar is exactness: the same chunk set, the same stored vectors with reuse on and off, and identical retrieval results. That is the check @davidh233's team already ran against their version.

Start in src/app/crawl/pipeline.rs and trace get_vectors_by_relative_paths through the incremental crawl and embedding worker. Review the sequencing constraints from issues #82, #84, and #89, then verify the implementation against the stated acceptance bar: bit-identical vectors, the same chunk set, and identical retrieval results with reuse enabled or disabled.