codeforpdx / codeforpdx/tenantfirstaid
LangSmith Experiments on Single-turn dataset
Open
@yangm2 is already working on this.
Since Jan 15, 2026.
backend
- Dominant language
- Python
- Stars
- 13
- Forks
- 21
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 3
Description
This issue captures ideas for experiments to run against single-turn baseline results
- gemini 2.5 pro w/o RAG - quantify how much RAG is improving correctness
- gemini 3.0 preview w/ RAG - quantify correctness/tokens-per-second/tone/cost-per-answer
- chatGPT frontier model (5.2?) w/ VertexSearch - gemini vs chatGPT
- thinking budget (in tokens) - investigate correlation factor between thinking budget and correctness, thinking budget and cost-per-answer
- gemini 2.5 pro w/ few-shot examples ??? - correctness
- fine-tune gemini 2.5 pro ??? - correctness
- a/b-test variations on system prompt - correctness
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.