Replace per-source extraction parsers with a single LLM path — measured, model-primary across all corpora

## Question

Can a local, CPU-bound LLM replace the per-source extraction parsers — deleting most of the
~6,195 lines / ~90 regexes in `sources/`, and surviving City reformatting that silently breaks
regex?

**Measured over two days. Answer: no for replacement, yes for drift robustness.** The two
findings are not in conflict; they describe different regimes.

## TL;DR

| | verdict |
|---|---|
| Replace extraction with an LLM | **No** — parity on 1 corpus of 3; would degrade the archive |
| LLM survives reformatting better than regex | **Yes, decisively** — −4% vs −88…−100% |
| Is CPU inference affordable here | **Yes** — ~5 min/night at real volume |
| GLiNER2 (extractive, 205M) | **No** — fabricates 64% of relations, undetectably |

## Setup

Qwen3-14B-Q4_K_M via llama.cpp, schema-constrained (outlines), 4 CPU threads, i3-13100 / 15 GB,
no GPU. Scored against **20,848 already-validated rows** across three deliberately contrasting
structures. Ground truth is each corpus's own validated parser output plus, where available, the
documents' own declared bid counts.

## Result 1 — drift robustness: the LLM wins decisively

Formatting mutations applied to the cell grid, both extractors given byte-identical input.
Mutations mimic plausible City reformats, not adversarial tricks.

| mutation | regex | LLM (14B) |
|---|---|---|
| baseline | 100% | **100%** (51/51) |
| header renamed (`Bidder`→`Vendor`) | 100% | 100% |
| columns reordered | **0%** | 96% |
| `$1,484,065.00` → `1,484,065.00 CAD` | **12%** | 100% |
| one extra column inserted | **0%** | 100% |

Three of four reformats take the parser to near-zero — and this is the **hardened** cell parser
from #151, not a strawman. Inserting a single column zeroes regex and costs the model nothing.

## Result 2 — replacement: the incumbent parsers win

Deterministic location (existing anchors) + LLM extraction, grouped per document:

| corpus | incumbent | LLM | note |
|---|---|---|---|
| `ep_board` | 100% | **99% / 98%** | parity |
| `award_summary` | 100% | **60% / 92%** | misses 40% of the cell parser's rows |
| `trca_board` | 100% | **45% / 49%** | half the emitted names are wrong |

`noloc` = 0 everywhere — **every failure is extraction, not retrieval**, which rules out
"better anchoring would fix it". Generic retrieval was also tried and was worse
(29%/56% on TRCA).

### Why — the important part

The value in these parsers is not the regex. It is the **documented exceptions**, each of which
cost real measurement to discover:

- #116 — the City leaves numbered row `5.` blank rather than omitting the bidder; an RFP lists
proponents with **no price at all**; a multi-package cell holds a whole column and must be
zipped positionally or refused.
- #94 — an OUTCOME in the price column (`Non-Compliant`) is still a bid.
- #87/#116 — `2489960 Ontario Inc.` is a real firm, not a leaked price.

A general prompt cannot reproduce these, and a larger model will not either: nothing in the
document says that a blank numbered row means an omission rather than an absence. **Capability
does not fix this; corpus knowledge does, and corpus knowledge is what the parsers ARE.**

## Result 3 — GLiNER2 rejected, and why the safety argument failed

GLiNER2 (Fastino, 205M, Apache 2.0, CPU-first) was tried first because it tags spans in the
input and therefore *cannot invent a value*. That property held and was **worthless alone**:

- verbatim verification failures: **0**
- (bidder, price) pairs correct: **29/80 — 36%**

Every name real, every price real, **64% of the relations between them fabricated**. It emitted
names in bullet-list order (alphabetical) and zipped them positionally against prices in table
order (ascending), assigning `$548,415` to a firm that was disqualified and had no price at all.

**A bid record is a relation, and a relation has no verbatim existence in the text to check
against.** This archive is almost entirely relations — bidder↔price, table↔solicitation,
award↔supplier. Any future local-model work must be scored on *relation* accuracy against
independent ground truth, never on whether extracted strings appear in the source.

## Cost, if we ever do adopt this

- **Steady state is tiny**: under a dozen new documents/day (Award Summary Forms ~30/month,
agency board reports ~400/year). At ~31 s/doc that is **~5 minutes a night** — a nightly that
already drives a headed Chromium will not notice. **Model choice is capped by RAM, not
throughput.**
- **Backlog is one-time**: 6,150 documents ≈ 53 h on this CPU, ≈ 6 h on an RTX 3060.
- **Constrained decoding is a hard requirement, not an optimisation.** Unconstrained, a 1.7B
emitted malformed JSON on the simplest table in the corpus. With reasoning enabled the same
model read that table correctly in prose — comprehension was never the bottleneck,
serialisation was.

## Proposed direction (not implemented)

1. **Keep the existing parsers as the primary path.** They measure better and encode discoveries.
2. **LLM as a drift FALLBACK** — fires only when a parser returns zero rows or fails its own
ground-truth check (a declared bid count it cannot satisfy). Near-zero cost, and it converts a
City reformat from a silent outage into a degraded-but-working night. This addresses the
actual motivation.
3. **LLM as the FIRST pass for NEW sources** — where the O(n)-per-source cost really lives. Every
new agency has needed a bespoke parser; start new ones on LLM + ground-truth check and write a
parser only where it measurably underperforms.

**Honest limitation: this does not reduce the existing ~6,195 lines**, which was the original
goal. The measurement does not support deleting them.

### If adopted, one hard constraint

`db.rebuild_agency_bids` re-derives every row for a source on each store pass — correct when
derivation is regex and costs milliseconds, ruinous when it is a 14B call. It would need a cache
keyed on the PDF's **`sha256` plus a model+prompt version stamp**, never a "skip if rows exist"
guard (which is what makes stale rows persist).

## Caveats on this evaluation

- Samples are 30 documents/corpus, not full corpora.
- **Unverified**: whether `award_summary`'s missing 40% actually clusters on the #116 exception
shapes. Cheap to check by diffing missed rows; it separates "buy a bigger model" from "the
exceptions do not transfer".
- **Four apparatus bugs occurred during this evaluation**, each producing a plausible-looking
number: a flattening bug returning exactly one row per document; unconstrained malformed JSON
scored as extraction (recall appeared to *rise* under corruption); a patch that silently failed
while relaunching stale code; and a grouping artifact that scored TRCA documents once per award
and counted the other awards' bidders as false positives. Three were caught only because a
number was *impossible* rather than merely wrong.

That is the most transferable finding here. **A general extraction layer needs ground-truth
invariants wired in as a runtime property, not an evaluation step** — the way `schema_check.py`
already works for feed drift. Without that, we would not discover the archive was wrong; it
would just get quieter.

Full working notes: `docs/superpowers/notes/2026-07-28-*` and `2026-07-29-*`.

Start with db.rebuild_agency_bids, schema_check.py, and the working notes in docs/superpowers/notes/2026-07-28-* and 2026-07-29-*. Define the parser-primary, LLM-fallback direction against the measured corpus results, including ground-truth checks and a cache keyed by PDF sha256 plus model and prompt versions; done means drift can fall back without replacing validated parser output.

Written by the indexing model from the issue text.