CivicTechTO / CivicTechTO/toronto-bids
Cached, versioned extractor seam — prefactor for model-based extraction
- Dominant language
- Python
- Stars
- 7
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
> *This was generated by AI during triage.*
## Parent
#205
## What to build
`rebuild_agency_bids` re-derives every `agency_bid` row from the held PDFs on each store pass — correct when derivation is regex and costs milliseconds, ruinous when it is a model call per document. This ticket adds a cache keyed on the document's **`sha256` plus a model+prompt version stamp**, so a document that has not changed and whose extraction prompt has not changed is never re-extracted.
The existing regex derivation moves behind this seam unchanged. 800 tests green, zero behaviour change. A second extractor (the model path) becomes a drop-in behind it.
**"Skip if rows exist" is the wrong guard** — that is what makes stale rows persist. The version stamp is what makes re-extraction trigger on a prompt change, not on whether the table is empty.
## Acceptance criteria
- [ ] A cache layer exists, keyed on `(sha256, extractor_version)`, that short-circuits extraction when the key matches
- [ ] Existing regex-based derivation produces identical `agency_bid` rows through the new seam
- [ ] A changed `extractor_version` causes re-extraction on the next store pass
- [ ] 800 existing tests pass unchanged
## Blocked by
None — can start immediately.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at rebuild_agency_bids and trace the store pass that derives agency_bid rows from held PDFs. Use the existing regex derivation and 800-test baseline to verify identical results, then confirm unchanged sha256 and extractor_version skips extraction while a changed version re-extracts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100