CivicTechTO / CivicTechTO/toronto-bids
A surrogate per-solicitation identity: our own key, City identifiers as attributes
- Dominant language
- Python
- Stars
- 7
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Came out of the #123 discussion. The archive has **four disjoint keyspaces** that deliberately don't join: `document_number` (competitive/Ariba), `workspace_number` (non-competitive), `call_number` (2009–2012 composite awards), and council `reference` (agenda items, staff reports, bids). CLAUDE.md is explicit that there is "no join between them."
## The idea
Give each procurement **our own surrogate primary key**, and demote every City-provided identifier to a nullable attribute/link on it:
```
procurement(
id INTEGER PRIMARY KEY, -- ours, stable
document_number TEXT NULL, -- Ariba, when known
workspace_number TEXT NULL, -- non-competitive, when known
call_number TEXT NULL, -- 2009–2012 composite, when known
council_reference TEXT NULL, -- when known
...
)
```
Everything (awards, bids, documents, postings) hangs off `id`. This makes representable:
- **A record known without any City identifier** (e.g. learned from a staff report or a document, no doc/workspace/call number). Not possible today — every source hands us *some* identifier — but a foundational identity model should allow it.
- **Identifier equivalence** — "this 2018 call-number council item and this later document-number spine row are the same procurement." That is exactly what #77 is trying to establish; today there is nowhere to *record* the answer.
## The hard part (why this needs its own scrutiny)
The surrogate key does **not** solve matching — it only gives you somewhere to store a match you can justify. Deciding *when two records are the same procurement* is the fuzzy problem #77 is bounded by. And this archive's repeated principle is **a wrong merge is worse than none**: a surrogate identity with an over-eager matcher behind it would fuse two distinct procurements under one id, which is worse than keeping them apart. The matching rules — and their false-positive rate — are the real design work here, not the schema.
## Scope / why it is not part of #123
This is a keyspace re-architecture: it touches every table, every join, the supplier-linking pass, and the export shape. #123 (per-solicitation content index) ships on `document_number` now and is **forward-compatible** — when this surrogate spine lands, the document index re-homes onto `id` with a mechanical migration, no rework of the extraction.
## What a design for this should settle
- The spine table and how each existing corpus maps onto it (including the ones that currently key on `reference` / `call_number` / `workspace_number`).
- Matching rules to populate the identifier columns, with a measured false-positive rate, following the calibrate-against-ground-truth discipline already used in #77 / #96.
- Migration: keep `document_number` etc. as the natural keys during ingest, resolve to `id` in a pass, rebuild FKs (same pattern as the supplier dimension).
- Export shape: does the public artifact expose `id`, and does it change the top-level sectioning (solicitations / noncompetitive / composite_awards / council_items become one entity?).
Related: #77 (identifier matching this would record), #96 (composite call-number keyspace), #123 (content index that re-homes onto this later).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with CLAUDE.md and the discussions in #77, #96, and #123 to understand the separate keyspaces and existing matching assumptions. Compare the proposed spine with the supplier-dimension migration pattern described in the issue. Done means the design settles corpus mapping, matching and false-positive measurement, migration, foreign keys, and public export shape.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100