CivicTechTO / CivicTechTO/toronto-bids
Committee/Council award bid capture — built, and bounded to ~1% by the doc-number join key
- Dominant language
- Python
- Stars
- 7
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
_Surfaced by the zero-bid investigation (#163). A distinct gap from the pre-Ariba linkage gap (#124)._
## The gap
**99 awarded solicitations, 2020+, each over $3M — totalling $3.11B — have zero captured bids and nothing else captured either**: no Ariba Discovery posting, no Award Summary Form, no staff report on file, no council item. Examples: `$197M District 2 Waste Collection`, `$139M`, `$137M Containerized Waste Collection`, `$126M construction`, `$96M General Contracting (Paramedic)`, `$91M`, `$90M Pax/RoPax Ferry Vessels`, `$89M Basement Flooding Protection`, `$80M sewer`, `$75M Ashbridges Bay`.
## Why they're missing
These awards exceed the Bid Award Panel's delegation ceiling, so they were decided by **Council or a Standing Committee** (Infrastructure, Executive, General Government, etc.), not the Bid Award Panel. The archive's bid capture (#84/#94) only mines **Bid Award Panel / Bid Committee** agendas (the 891 cached BA/BD pages), so these never enter the corpus. The award itself comes from the OData spine; the *bid table* is not.
Few items, but **disproportionate dollar value** ($3.11B) — the exact awards where "was this major spending competitive?" matters most.
## Where the bids likely live
Council/committee **staff reports on legdocs** (plain HTTP, not Akamai-gated — same host we already fetch `background_pdf` staff reports from). A committee report recommending a large award typically tabulates the bids received, the same way Bid Award Panel reports do.
## Suggested direction
1. Identify the target set: awarded solicitations over the panel threshold with no captured bids (the 99 here; refresh the query as new awards land).
2. Locate each award's committee decision + staff report. The committee agendas are on TMMIS (Akamai-gated → headed browser, like the existing council/agency scrapers); the staff-report PDFs are plain HTTP.
3. Parse the report's bid table (a new parser — committee reports differ in layout from BA agendas; validate against real reports per the #136/#138 discipline).
4. **Caveat — some are genuine zero:** a handful are emergency / sole-source / "only supplier" (e.g. the $96M Paramedic General Contracting is flagged "emergency"). Refuse those (no bids by nature), don't invent them.
## Scope note
This is a **new capture source**, larger than a linking pass — distinct from #124 (which links *already-captured* pre-Ariba bids). Bounded and high-value; could be scoped to just the >$3M / >$10M tier first.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the existing bid-capture flow and the background_pdf staff-report fetches, then inspect TMMIS committee agendas for the over-$3M awarded solicitations with no captured bids. Compare real committee reports with Bid Award Panel layouts and follow the validation discipline referenced in #136/#138. Done means committee bid tables are captured while emergency, sole-source, and other genuine zero-bid awards are excluded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100