codeforpdx / codeforpdx/tenantfirstaid
Add web-scraper RAG tool to create_app_gcs for Oregon Law Library ingestion
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13
- Forks
- 21
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 3
Description
Follow-up from https://github.com/codeforpdx/tenantfirstaid/pull/349. yangm2 noted that create_app_gcs currently attaches 1 datastore to an app, but apps can have 0+ datastores when web-site scrapers are configured. The Oregon Law Library is the motivating source.
Scope
Add a --scraper-url (repeatable) argument to backend/scripts/create_app_gcs.py that registers a web-scraper data source on the app in addition to (or instead of) the GCS-backed datastore.
Wire the scraper config through DocumentProcessingConfig / engine creation so the app indexes the crawled site alongside the corpus.
Document the URL patterns used for the Oregon Law Library and any rate-limit / robots.txt considerations.
Coordinate with Draft 2 — scraper targets are a natural fit for the TOML config.
Acceptance
make create-app-gcs SCRAPER_URL=... (or equivalent) creates an app with both the GCS datastore and a web-scraper data source.
At least one scraper target (Oregon Law Library) is configured and verified to return results in the RAG tool.
Refs: https://github.com/codeforpdx/tenantfirstaid/pull/349
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with backend/scripts/create_app_gcs.py and trace how DocumentProcessingConfig and engine creation currently attach the GCS datastore. Then inspect the make create-app-gcs entry point and the TOML configuration referenced by Draft 2. Done means a repeatable SCRAPER_URL creates an app with the configured scraper and optional GCS datastore, with an Oregon Law Library target documented and verified to return RAG results while respecting rate limits and robots.txt.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100