codeforpdx / codeforpdx/tenantfirstaid

Add web-scraper RAG tool to create_app_gcs for Oregon Law Library ingestion

Open
#355 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13
Forks
21
Avg merge
6d 7h
Merged PRs (30d)
3

Description

Follow-up from https://github.com/codeforpdx/tenantfirstaid/pull/349. yangm2 noted that create_app_gcs currently attaches 1 datastore to an app, but apps can have 0+ datastores when web-site scrapers are configured. The Oregon Law Library is the motivating source.

Scope

Add a --scraper-url (repeatable) argument to backend/scripts/create_app_gcs.py that registers a web-scraper data source on the app in addition to (or instead of) the GCS-backed datastore.
Wire the scraper config through DocumentProcessingConfig / engine creation so the app indexes the crawled site alongside the corpus.
Document the URL patterns used for the Oregon Law Library and any rate-limit / robots.txt considerations.
Coordinate with Draft 2 — scraper targets are a natural fit for the TOML config.
Acceptance

make create-app-gcs SCRAPER_URL=... (or equivalent) creates an app with both the GCS datastore and a web-scraper data source.
At least one scraper target (Oregon Law Library) is configured and verified to return results in the RAG tool.
Refs: https://github.com/codeforpdx/tenantfirstaid/pull/349

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with backend/scripts/create_app_gcs.py and trace how DocumentProcessingConfig and engine creation currently attach the GCS datastore. Then inspect the make create-app-gcs entry point and the TOML configuration referenced by Draft 2. Done means a repeatable SCRAPER_URL creates an app with the configured scraper and optional GCS datastore, with an Oregon Law Library target documented and verified to return RAG results while respecting rate limits and robots.txt.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.