CivicTechTO / CivicTechTO/toronto-bids

Validate the provisional bids&tenders portal parser once a portal has data

Open
#140 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
7
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Follow-up from #135 / PR #139. The bids&tenders portal-listing capture is merged and armed, but `parse_listing` shipped **PROVISIONAL**: both TRCA and Zoo portals are currently empty (`total=0`, verified live), so it is mapped to the portal's grid-JS field names and validated only against a **synthetic** fixture (`tests/fixtures/agencies/bids_tenders_record_sample.json`, labelled synthetic in SOURCES.md).

**Trigger:** when a TRCA or Zoo bid first appears on its portal (check with `tb enrich-agencies --portal --record` — non-zero output means data exists).

**Then:**
1. `--record` captures the real JSON to `/agencies/portal_recordings/`; replace the synthetic fixture with a real one.
2. Fix `parse_listing` against real records and re-validate with recall **measured** against the endpoint's `total` (the #136 lesson — offline fixtures lied about real corpora three times; don't trust a synthetic one).
3. Settle the deferred questions the empty portals left open:
- the exact `status` code → Open/Awarded/Cancelled mapping;
- whether the portal bid number matches the board-report `native_ref` format (how often the COALESCE enrich fires vs. a distinct row);
- whether awarded records expose a supplier/value → build a portal `agency_award` path (currently not built);
- whether `store_listings` should overwrite `status` on a nightly re-fetch (it currently backfills with `overwrite=False`, so a stale status is not refreshed — the deliberate no-data-loss choice, revisit with real data).

Until then the nightly portal step is a verified clean no-op. Spec: `docs/superpowers/specs/2026-07-18-bidsandtenders-portal-listings-design.md` ("Deferred to first-real-data").

Contributor guide

No contributing guide indexed for this repository

Research direction

Wait until `tb enrich-agencies --portal --record` reports data for TRCA or Zoo, then capture a real record in `/agencies/portal_recordings/`. Read `docs/superpowers/specs/2026-07-18-bidsandtenders-portal-listings-design.md` and compare the real capture with `parse_listing` and `tests/fixtures/agencies/bids_tenders_record_sample.json`. Done means validation against the endpoint’s `total` and resolution of the deferred mapping, matching, award-data, and status-refresh questions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.