Retire the per-file sentinel mechanism

Tracking issue for BL48, since @davidh233's team recently encountered an issue that justifies prioritizing it.

Each chunk row carries a file_complete flag, and chunk 1 of every file acts as a sentinel that flips true only once all of the file's chunks are durably written. That design comes from an earlier storage backend with no tables, where cross-row atomicity had to be encoded in row content. LanceDB has manifest-version atomicity, so if the writer batches all of a file's chunks into a single merge_insert, the write is already all-or-nothing and "file is fully indexed" collapses to "any chunk row exists for this file_id."

Why now

The trigger recorded on the backlog item was sentinel mechanics costing readability or correctness on a touching change. What arrived instead was field data, which is better. Two of the three problems reported in a large-monorepo evaluation are the sentinel's cost rather than incidental to it:

• Marking files complete at each checkpoint runs one full-table scan and one commit per file, roughly 91 of each per checkpoint. This OOM-killed a 48k-file crawl and drove the version count to 11,856. Batching it is issue #83, which reduces those to one scan and one commit per batch. Retiring the sentinel removes the write pass altogether, because a file that has rows is complete by construction.
• Classify reads every file's sentinel on every re-crawl to decide skip versus redo. On a 230k-file repository that is a fixed cost paid every round regardless of how little changed.

So #83 is scaffolding that this issue deletes.

Scope

• Buffer a file's chunks in the writer thread and write them in one merge_insert call.
• Remove file_complete from the schema and update the predicates that read it.
• Replace the completion check with an existence check on file_id.

The durability property is the part that needs care, not the lookup. Today chunk 1 flipping true is what tells a resumed crawl that the previous run finished writing that file. Under the new shape that guarantee comes from the single-call write being atomic at the manifest level, so the buffering is the mechanism, not an optimization on top of it. Partial-write states that are representable today (some chunks present, sentinel false) stop being representable, which is the point, but any recovery logic that reads those states has to be retired with them rather than left checking a distinction that no longer exists.

One thing that looks like a durability regression and is not: today a file's chunks can span two checkpoints, so an interrupted crawl leaves partially-written files on disk, while the buffered writer would lose them. But those partial rows never help a resume. A file whose sentinel is false fails the fast-path check and goes back through chunking and embedding in full, so the resumed crawl re-does that work either way. Buffering gives up nothing a resume could use.

What the fast-path check becomes

Today classify skips a file when its sentinel row exists, file_complete is true, and, when vector is in the label's selection, the sentinel row's vector is non-NULL. That last clause survives retirement: an FTS-only crawl still writes a file's rows with NULL vectors, so row existence alone cannot prove vectors exist. The new predicate is that the row {file_id}:1 exists, plus the same vector clause. Ordinal 1 always exists once a file is written, because the chunker renumbers ordinals after dropping empty chunks, so the probe stays a row_id point lookup and the batched read in #85 and the BTree in #89 apply to it unchanged.

This does not make classify cheaper by itself. The probe is still one read per file, in #85's batched shape. What retirement deletes is the write pass and the flag semantics, not the read.

A correctness improvement comes along for free, though. docs/design/crawl.md documents a transient gap where an interrupted vector crawl followed by an FTS-only crawl can leave a file complete with vectors on some chunks and not others. With per-file atomic writes, a file's vectors arrive in one call or not at all, so that state stops being constructible. This issue is the structural separation that document promises.

The writer buffers per file

Today the writer flushes whatever chunks have accumulated every 60 seconds. Under the new shape it holds each file's chunks until the last one arrives, and a checkpoint writes only whole files, several per merge_insert call. The FTS-only upsert path already works this way, writing whole files per call and asserting complete chunk sets, so the change is confined to the vector path. Commit cadence stays at the checkpoint, and the completion-mark commits disappear, so versions per checkpoint drop from one plus one per completed file to one.

The buffer's memory is bounded by the chunks in flight between checkpoints. At the measured 1.4 to 4.6 chunks/s that is a few hundred chunks of text plus 3KB vectors, a few MB, with the worst case being a single pathological file whose chunks all wait for its slowest one. Batch boundaries must align to file boundaries: a batch may hold several whole files but never part of one, and a file with more chunks than UPSERT_BATCH_SIZE gets one oversized batch, since splitting it would reintroduce the partial state this issue exists to remove.

Sequencing

Removing file_complete from the schema bumps MONODEX_SCHEMA_VERSION, so existing databases are rejected and the remedy under the pre-1.0 policy is init-db --delete-everything plus a re-crawl. #82 bumps EMBEDDER_ID and #90 adds a column, so the three should ship together as one rebuild rather than three. #83 goes first and stays useful for as long as the column exists, and #85 also goes ahead of this, since re-crawls are unusable at scale in the meantime and this issue has the longer investigation.

Start with the vector writer, classify fast-path, and schema handling described in the issue, then read docs/design/crawl.md for the durability invariant. Buffer whole files, remove file_complete and its completion-mark commits, and use the {file_id}:1 existence probe with the existing vector condition. Verify schema-version handling and the init-db --delete-everything rebuild path; done means writes are atomic per file and partial file states are no longer used.