CivicTechTO / CivicTechTO/toronto-bids

A surrogate per-solicitation identity: our own key, City identifiers as attributes

Open
#124 3 comments 0 reactions 0 assignees View on GitHub
enhancement ready-for-human
Dominant language
Python
Stars
7
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Came out of the #123 discussion. The archive has **four disjoint keyspaces** that deliberately don't join: `document_number` (competitive/Ariba), `workspace_number` (non-competitive), `call_number` (2009–2012 composite awards), and council `reference` (agenda items, staff reports, bids). CLAUDE.md is explicit that there is "no join between them."

## The idea

Give each procurement **our own surrogate primary key**, and demote every City-provided identifier to a nullable attribute/link on it:

```
procurement(
id INTEGER PRIMARY KEY, -- ours, stable
document_number TEXT NULL, -- Ariba, when known
workspace_number TEXT NULL, -- non-competitive, when known
call_number TEXT NULL, -- 2009–2012 composite, when known
council_reference TEXT NULL, -- when known
...
)
```

Everything (awards, bids, documents, postings) hangs off `id`. This makes representable:
- **A record known without any City identifier** (e.g. learned from a staff report or a document, no doc/workspace/call number). Not possible today — every source hands us *some* identifier — but a foundational identity model should allow it.
- **Identifier equivalence** — "this 2018 call-number council item and this later document-number spine row are the same procurement." That is exactly what #77 is trying to establish; today there is nowhere to *record* the answer.

## The hard part (why this needs its own scrutiny)

The surrogate key does **not** solve matching — it only gives you somewhere to store a match you can justify. Deciding *when two records are the same procurement* is the fuzzy problem #77 is bounded by. And this archive's repeated principle is **a wrong merge is worse than none**: a surrogate identity with an over-eager matcher behind it would fuse two distinct procurements under one id, which is worse than keeping them apart. The matching rules — and their false-positive rate — are the real design work here, not the schema.

## Scope / why it is not part of #123

This is a keyspace re-architecture: it touches every table, every join, the supplier-linking pass, and the export shape. #123 (per-solicitation content index) ships on `document_number` now and is **forward-compatible** — when this surrogate spine lands, the document index re-homes onto `id` with a mechanical migration, no rework of the extraction.

## What a design for this should settle

- The spine table and how each existing corpus maps onto it (including the ones that currently key on `reference` / `call_number` / `workspace_number`).
- Matching rules to populate the identifier columns, with a measured false-positive rate, following the calibrate-against-ground-truth discipline already used in #77 / #96.
- Migration: keep `document_number` etc. as the natural keys during ingest, resolve to `id` in a pass, rebuild FKs (same pattern as the supplier dimension).
- Export shape: does the public artifact expose `id`, and does it change the top-level sectioning (solicitations / noncompetitive / composite_awards / council_items become one entity?).

Related: #77 (identifier matching this would record), #96 (composite call-number keyspace), #123 (content index that re-homes onto this later).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with CLAUDE.md and the discussions in #77, #96, and #123 to understand the separate keyspaces and existing matching assumptions. Compare the proposed spine with the supplier-dimension migration pattern described in the issue. Done means the design settles corpus mapping, matching and false-positive measurement, migration, foreign keys, and public export shape.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.