BuildCanada / BuildCanada/CanadaSpends

Consolidate spending items scraping into this repo

Open
#61 1 comment 0 reactions 0 assignees View on GitHub
Granular Spending Platform
Dominant language
TypeScript
Stars
84
Forks
35
Avg merge
5h 30m
Merged PRs (30d)
4

Description

Currently all scraping underlying the search functionality is not running on a regular basis and requires a lot of manual work to publish a new release.

- Create database & tables for scraped data
- Likely a core table with JSONB for source-specific data.
- Need to support:
- Global Affairs Canada

Can be split into two parts:
- Scraping infrastructure (database/cron jobs)
- Search Infrastructure (meillisearch, currently it is backed by typesense, but typesense is more expensive to host because it loads everything into memory)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.