CivicTechTO / CivicTechTO/toronto-bids

Committee/Council award bid capture — built, and bounded to ~1% by the doc-number join key

Open
#164 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
7
Forks
2
PR merge metrics
No merged PRs in 30d

Description

_Surfaced by the zero-bid investigation (#163). A distinct gap from the pre-Ariba linkage gap (#124)._

## The gap

**99 awarded solicitations, 2020+, each over $3M — totalling $3.11B — have zero captured bids and nothing else captured either**: no Ariba Discovery posting, no Award Summary Form, no staff report on file, no council item. Examples: `$197M District 2 Waste Collection`, `$139M`, `$137M Containerized Waste Collection`, `$126M construction`, `$96M General Contracting (Paramedic)`, `$91M`, `$90M Pax/RoPax Ferry Vessels`, `$89M Basement Flooding Protection`, `$80M sewer`, `$75M Ashbridges Bay`.

## Why they're missing

These awards exceed the Bid Award Panel's delegation ceiling, so they were decided by **Council or a Standing Committee** (Infrastructure, Executive, General Government, etc.), not the Bid Award Panel. The archive's bid capture (#84/#94) only mines **Bid Award Panel / Bid Committee** agendas (the 891 cached BA/BD pages), so these never enter the corpus. The award itself comes from the OData spine; the *bid table* is not.

Few items, but **disproportionate dollar value** ($3.11B) — the exact awards where "was this major spending competitive?" matters most.

## Where the bids likely live

Council/committee **staff reports on legdocs** (plain HTTP, not Akamai-gated — same host we already fetch `background_pdf` staff reports from). A committee report recommending a large award typically tabulates the bids received, the same way Bid Award Panel reports do.

## Suggested direction

1. Identify the target set: awarded solicitations over the panel threshold with no captured bids (the 99 here; refresh the query as new awards land).
2. Locate each award's committee decision + staff report. The committee agendas are on TMMIS (Akamai-gated → headed browser, like the existing council/agency scrapers); the staff-report PDFs are plain HTTP.
3. Parse the report's bid table (a new parser — committee reports differ in layout from BA agendas; validate against real reports per the #136/#138 discipline).
4. **Caveat — some are genuine zero:** a handful are emergency / sole-source / "only supplier" (e.g. the $96M Paramedic General Contracting is flagged "emergency"). Refuse those (no bids by nature), don't invent them.

## Scope note

This is a **new capture source**, larger than a linking pass — distinct from #124 (which links *already-captured* pre-Ariba bids). Bounded and high-value; could be scoped to just the >$3M / >$10M tier first.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the existing bid-capture flow and the background_pdf staff-report fetches, then inspect TMMIS committee agendas for the over-$3M awarded solicitations with no captured bids. Compare real committee reports with Bid Award Panel layouts and follow the validation discipline referenced in #136/#138. Done means committee bid tables are captured while emergency, sole-source, and other genuine zero-bid awards are excluded.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.