CivicTechTO / CivicTechTO/toronto-bids

Cached, versioned extractor seam — prefactor for model-based extraction

Open
#207 0 comments 0 reactions 0 assignees View on GitHub
enhancement ready-for-agent
Dominant language
Python
Stars
7
Forks
2
PR merge metrics
No merged PRs in 30d

Description

> *This was generated by AI during triage.*

## Parent

#205

## What to build

`rebuild_agency_bids` re-derives every `agency_bid` row from the held PDFs on each store pass — correct when derivation is regex and costs milliseconds, ruinous when it is a model call per document. This ticket adds a cache keyed on the document's **`sha256` plus a model+prompt version stamp**, so a document that has not changed and whose extraction prompt has not changed is never re-extracted.

The existing regex derivation moves behind this seam unchanged. 800 tests green, zero behaviour change. A second extractor (the model path) becomes a drop-in behind it.

**"Skip if rows exist" is the wrong guard** — that is what makes stale rows persist. The version stamp is what makes re-extraction trigger on a prompt change, not on whether the table is empty.

## Acceptance criteria

- [ ] A cache layer exists, keyed on `(sha256, extractor_version)`, that short-circuits extraction when the key matches
- [ ] Existing regex-based derivation produces identical `agency_bid` rows through the new seam
- [ ] A changed `extractor_version` causes re-extraction on the next store pass
- [ ] 800 existing tests pass unchanged

## Blocked by

None — can start immediately.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at rebuild_agency_bids and trace the store pass that derives agency_bid rows from held PDFs. Use the existing regex derivation and 800-test baseline to verify identical results, then confirm unchanged sha256 and extractor_version skips extraction while a changed version re-extracts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.