codeforpdx / codeforpdx/tenantfirstaid
Multi-bucket / multi-jurisdiction datastore support across GCS ingestion pipeline
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13
- Forks
- 21
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 3
Description
Follow-up from https://github.com/codeforpdx/tenantfirstaid/pull/349 (deferred per https://github.com/codeforpdx/tenantfirstaid/pull/349#pullrequestreview- and leekahung's reply).
The current GCS provisioning flow assumes a single bucket → single datastore → single app. To support adding jurisdictions (ORS / Portland / Eugene / future) without coupling them into one bucket, the pipeline should support an M:1 bucket-to-datastore relationship.
Scope
backend/Makefile: wrap upload-to-gcs / create-datastore-gcs targets in a foreach over a jurisdiction list (referenced at backend/Makefile:70 in https://github.com/codeforpdx/tenantfirstaid/pull/349).
backend/scripts/create_datastore_gcs.py: accept multiple --bucket arguments (or repeated --bucket) and attach all of them to the new datastore. The check_bucket_location_compat pre-flight check should run per bucket.
backend/scripts/generate_metadata_jsonl.py: emit per-jurisdiction metadata.jsonl files into per-jurisdiction directories instead of a single combined file. This also simplifies filtering and lets each jurisdiction's doc structure evolve independently.
Document the jurisdiction list source of truth (likely the corpus-cfg.TOML proposed in the sibling issue).
Acceptance
make upload-to-gcs / make create-datastore-gcs can provision N buckets attached to a single datastore from one invocation.
Per-jurisdiction metadata.jsonl files are kept separate end-to-end.
Existing single-bucket flow still works (sensible default when only one jurisdiction is configured).
Refs: https://github.com/codeforpdx/tenantfirstaid/issues/347, https://github.com/codeforpdx/tenantfirstaid/pull/349
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with backend/Makefile around line 70 and inspect the upload-to-gcs and create-datastore-gcs targets, then read backend/scripts/create_datastore_gcs.py and generate_metadata_jsonl.py. Trace how the jurisdiction list is sourced, including the proposed corpus-cfg.TOML in issue #347, and review PR #349's deferred approach. Done means one invocation provisions N buckets on one datastore, per-jurisdiction metadata remains separate end-to-end, and the single-jurisdiction default still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- backend, cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100