Revision-controlled corpus-cfg.TOML for RAG deployment + chunking/granularity flags

Open
#354 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
38/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
ai, backend, cloud, data

Research direction

Read #349, then inspect backend/generate_metadata_jsonl.py, backend/create_datastore_gcs.py, backend/create_app_gcs.py, and constants.py to map the existing arguments and location setting. Verify how the proposed TOML schema should represent each RAG tool and how CLI overrides interact with it. Done means the documented corpus-cfg.TOML is consumed by default, the listed flags work as overrides, and --datastore-id is repeatable.

Written by the indexing model from the issue text.

Description

Follow-up from #349. yangm2 raised that as we accumulate multiple RAG tools with unique settings, reproducible deployment should be driven by a checked-in config file rather than ad-hoc CLI args.

Proposed config (backend/corpus-cfg.TOML or similar): revision-controlled settings for each RAG tool — bucket list, datastore ID, app ID, chunking strategy, granularity, annotation handling, location.

RAG experimentation flags to wire up (from yangm2's review table in #349):

Impact Experiment Script Arg
Highest Section-boundary chunking generate_metadata_jsonl.py --granularity [file|section] — switches from one entry per .txt file to one entry per 90.XXX section
High Cross-reference expansion generate_metadata_jsonl.py --expand-refs — resolves inline ORS X.XXX citations, appends target section text to each entry's struct_data
Medium Chunk size / overlap create_datastore_gcs.py --chunk-size INT and --chunk-overlap INT passed into DocumentProcessingConfig (pairs with --granularity file)
Medium Annotations datastore separation generate_metadata_jsonl.py + create_datastore_gcs.py --exclude-annotations on metadata generation; run twice to produce two metadata.jsonl files; --datastore-id on create_app_gcs.py made repeatable (action="append") so both datastores attach to one app

Most of these naturally live in the TOML once it exists, so they should land together (or the TOML should land first and these flags become overrides).

Also covers yangm2's comment on DEFAULT_VERTEX_AI_SEARCH_LOCATION in constants.py — long term that constant should be sourced from the TOML rather than a Python constant.

Acceptance

  • corpus-cfg.TOML exists in the repo with documented schema.
  • generate_metadata_jsonl.py, create_datastore_gcs.py, create_app_gcs.py consume the TOML by default; CLI flags above act as overrides.
  • --datastore-id on create_app_gcs.py is repeatable.

Refs: #349

Dominant language
Python
Stars
13
Forks
21
Avg merge
6d 7h
Merged PRs (30d)
3

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from codeforpdx/tenantfirstaid

All issues in codeforpdx/tenantfirstaid

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.