Revision-controlled corpus-cfg.TOML for RAG deployment + chunking/granularity flags
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 38/100
Research direction
Read #349, then inspect backend/generate_metadata_jsonl.py, backend/create_datastore_gcs.py, backend/create_app_gcs.py, and constants.py to map the existing arguments and location setting. Verify how the proposed TOML schema should represent each RAG tool and how CLI overrides interact with it. Done means the documented corpus-cfg.TOML is consumed by default, the listed flags work as overrides, and --datastore-id is repeatable.
Written by the indexing model from the issue text.
Description
Follow-up from #349. yangm2 raised that as we accumulate multiple RAG tools with unique settings, reproducible deployment should be driven by a checked-in config file rather than ad-hoc CLI args.
Proposed config (
backend/corpus-cfg.TOMLor similar): revision-controlled settings for each RAG tool — bucket list, datastore ID, app ID, chunking strategy, granularity, annotation handling, location.RAG experimentation flags to wire up (from yangm2's review table in #349):
Impact Experiment Script Arg Highest Section-boundary chunking generate_metadata_jsonl.py--granularity [file|section]— switches from one entry per.txtfile to one entry per90.XXXsectionHigh Cross-reference expansion generate_metadata_jsonl.py--expand-refs— resolves inline ORS X.XXX citations, appends target section text to each entry'sstruct_dataMedium Chunk size / overlap create_datastore_gcs.py--chunk-size INTand--chunk-overlap INTpassed intoDocumentProcessingConfig(pairs with--granularity file)Medium Annotations datastore separation generate_metadata_jsonl.py+create_datastore_gcs.py--exclude-annotationson metadata generation; run twice to produce twometadata.jsonlfiles;--datastore-idoncreate_app_gcs.pymade repeatable (action="append") so both datastores attach to one appMost of these naturally live in the TOML once it exists, so they should land together (or the TOML should land first and these flags become overrides).
Also covers yangm2's comment on
DEFAULT_VERTEX_AI_SEARCH_LOCATIONinconstants.py— long term that constant should be sourced from the TOML rather than a Python constant.Acceptance
corpus-cfg.TOMLexists in the repo with documented schema.generate_metadata_jsonl.py,create_datastore_gcs.py,create_app_gcs.pyconsume the TOML by default; CLI flags above act as overrides.--datastore-idoncreate_app_gcs.pyis repeatable.Refs: #349
- Dominant language
- Python
- Stars
- 13
- Forks
- 21
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 3
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from codeforpdx/tenantfirstaid
-
2026 roadmap Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
codeforpdx/tenantfirstaid#390 ·
-
backend good first issue infrastructure
Difficulty 3/5 1-2 days Newbie friendliness 48/100
codeforpdx/tenantfirstaid#389 ·
-
AI Prompts backend external-RAG infrastructure
Difficulty 4/5 3-5 days Newbie friendliness 45/100
codeforpdx/tenantfirstaid#372 · 2 comments ·
-
bug frontend
Difficulty 4/5 3-5 days Newbie friendliness 38/100
codeforpdx/tenantfirstaid#371 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
codeforpdx/tenantfirstaid#355 ·
All issues in codeforpdx/tenantfirstaid
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100