codeforpdx / codeforpdx/tenantfirstaid

Multi-bucket / multi-jurisdiction datastore support across GCS ingestion pipeline

Open
#353 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13
Forks
21
Avg merge
6d 7h
Merged PRs (30d)
3

Description

Follow-up from https://github.com/codeforpdx/tenantfirstaid/pull/349 (deferred per https://github.com/codeforpdx/tenantfirstaid/pull/349#pullrequestreview- and leekahung's reply).

The current GCS provisioning flow assumes a single bucket → single datastore → single app. To support adding jurisdictions (ORS / Portland / Eugene / future) without coupling them into one bucket, the pipeline should support an M:1 bucket-to-datastore relationship.

Scope

backend/Makefile: wrap upload-to-gcs / create-datastore-gcs targets in a foreach over a jurisdiction list (referenced at backend/Makefile:70 in https://github.com/codeforpdx/tenantfirstaid/pull/349).
backend/scripts/create_datastore_gcs.py: accept multiple --bucket arguments (or repeated --bucket) and attach all of them to the new datastore. The check_bucket_location_compat pre-flight check should run per bucket.
backend/scripts/generate_metadata_jsonl.py: emit per-jurisdiction metadata.jsonl files into per-jurisdiction directories instead of a single combined file. This also simplifies filtering and lets each jurisdiction's doc structure evolve independently.
Document the jurisdiction list source of truth (likely the corpus-cfg.TOML proposed in the sibling issue).
Acceptance

make upload-to-gcs / make create-datastore-gcs can provision N buckets attached to a single datastore from one invocation.
Per-jurisdiction metadata.jsonl files are kept separate end-to-end.
Existing single-bucket flow still works (sensible default when only one jurisdiction is configured).
Refs: https://github.com/codeforpdx/tenantfirstaid/issues/347, https://github.com/codeforpdx/tenantfirstaid/pull/349

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with backend/Makefile around line 70 and inspect the upload-to-gcs and create-datastore-gcs targets, then read backend/scripts/create_datastore_gcs.py and generate_metadata_jsonl.py. Trace how the jurisdiction list is sourced, including the proposed corpus-cfg.TOML in issue #347, and review PR #349's deferred approach. Done means one invocation provisions N buckets on one datastore, per-jurisdiction metadata remains separate end-to-end, and the single-jurisdiction default still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
backend, cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.