CivicTechTO / CivicTechTO/toronto-bids

Extraction client — text in, validated records out, refuse on malformed

Open
#208 0 comments 0 reactions 0 assignees View on GitHub
enhancement ready-for-agent
Dominant language
Python
Stars
7
Forks
2
PR merge metrics
No merged PRs in 30d

Description

> *This was generated by AI during triage.*

## Parent

#205

## What to build

A client module that takes document text and returns validated extraction records conforming to the schema in `docs/protocols/extraction-schema-by-document-type.md`. Two models: **Nemotron 3 Ultra (free)** as default, **GPT-5.6-Luna (~$4 for the full backlog)** as automatic fallback on rate-limit or unavailability. Via OpenRouter.

Four things that already bit during evaluation and must be handled:

1. **Retry with backoff.** Its absence scored Sol at 22% instead of 100% — four instant retries under rate limiting, silently returning empty.
2. **Validate-and-refuse for unconstrained models.** Nemotron has no structured-output support; it produces JSON by following instructions. Output must be schema-validated and a document refused rather than partially stored on malformed output.
3. **Provider quirks.** DeepSeek needs the literal word "json" in the prompt; Tencent ignores response schemas and returns markdown. Any production path needs that fallback logic or it will score a capable model as useless.
4. **Placeholder-credential guard.** `scrapers/.env.example` ships with `OPENROUTER_API_KEY=your-openrouter-api-key`; a checkout that copied it but never filled it in must fail immediately with a clear message, not 30 seconds later inside an HTTP call (#184 pattern).

`tb extract --dry-run ` prints validated records for a held document and writes nothing. Offline tests from recorded response fixtures (no network in tests — same discipline as the rest of the suite).

## Acceptance criteria

- [ ] `tb extract --dry-run ` produces validated records from a held document
- [ ] Nemotron default, Luna fallback on rate-limit or unavailability
- [ ] Malformed output is refused, not partially stored
- [ ] Placeholder API key detected and refused before any HTTP call
- [ ] Offline test suite using recorded response fixtures, no network
- [ ] `OPENROUTER_API_KEY` read from `scrapers/.env`, gitignored, never committed

## Blocked by

None — can start immediately (parallel with the extractor seam).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with docs/protocols/extraction-schema-by-document-type.md and inspect the existing extractor seam referenced by the parent issue. Review scrapers/.env.example and the #184 credential-guard pattern, then run the held-document tb extract --dry-run command and examine recorded response fixtures. Done means validated records are printed, malformed output and placeholder credentials are refused, fallback and retry behavior work, and tests make no network calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, cli, security, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.