Text Analysis Tutorial: RAG Basics (indexing, retrieval, prompt construction; small corpus)

---

# Title & Overview

**Template:** *RAG Basics: An Intermediate, End-to-End Analysis Tutorial*
**Overview (≤2 sentences):** Learners will implement a Retrieval-Augmented Generation (RAG) pipeline with indexing, retrieval, and prompt construction over a small corpus. It is intermediate because it emphasizes reproducibility, retrieval–generation trade-offs, and structured evaluation of retrieval quality and answer correctness.

# Purpose

The value-add is teaching learners how to **combine retrieval with generation** in a defensible way. They will explore indexing strategies, prompt design, and evaluation of RAG systems, while stressing reproducibility and guardrails for small-scale deployment.

# Prerequisites

* Skills: Python, Git, pandas, ML basics.
* NLP: embeddings, retrieval, generation models, evaluation metrics.
* Tooling: pandas, scikit-learn, Hugging Face Transformers, Sentence-Transformers, FAISS, Haystack, MLflow, FastAPI.

# Setup Instructions

* Environment: Conda/Poetry (Python 3.11), deterministic seeds.
* Install: pandas, scikit-learn, Hugging Face Transformers + Datasets, Sentence-Transformers, FAISS, Haystack, MLflow, FastAPI.
* Datasets:

* **Small:** WikiMovies (QA corpus).
* **Medium:** HotpotQA (subset for multi-hop retrieval).
* Repo layout:

```
tutorials/t11-rag-basics/
├─ notebooks/
├─ src/
│ ├─ indexer.py
│ ├─ retriever.py
│ ├─ generator.py
│ ├─ pipeline.py
│ └─ config.yaml
├─ data/README.md
├─ reports/
└─ tests/
```

# Core Concepts

* **RAG workflow:** corpus indexing → document retrieval → prompt construction → generation.
* **Indexing choices:** exact vs approximate nearest neighbors (FAISS).
* **Prompt construction:** retrieved docs + query → model input; guardrails to prevent overflow.
* **Evaluation:** retrieval precision\@k, generation accuracy/F1, calibration of confidence.
* **Error slicing:** by query type, document length, retrieval difficulty.

# Step-by-Step Walkthrough

1. **Data intake & preprocessing:** load WikiMovies/HotpotQA subsets; reproducible splits.
2. **Indexing:** build FAISS index on sentence/document embeddings (Sentence-Transformers).
3. **Retrieval baseline:** BM25 vs dense ANN embeddings; compare retrieval quality.
4. **Generation baseline:** DistilBART/T5-small with retrieved docs as context.
5. **Evaluation:** retrieval precision\@k, MRR; generation EM/F1; slice performance by query type.
6. **Error analysis:** missed retrievals, hallucinations, overly long prompts.
7. **Reporting:** metrics tables, retrieval/generation error cases → `reports/t11-rag-basics.md`.
8. *(Optional)* Serve: FastAPI endpoint with retrieval + generation pipeline, schema validation, max token guardrails.

# Hands-On Exercises

* Ablations: BM25 vs FAISS; top-k=1 vs top-k=5 retrieved docs.
* Robustness: add distractor documents; measure retrieval/generation accuracy.
* Slice analysis: compare single-hop vs multi-hop queries.
* Stretch: hybrid retrieval (BM25 + embeddings reranker).

# Common Pitfalls & Troubleshooting

* **Retriever failure cascades:** poor retrieval → bad generation.
* **Long prompts:** exceeding model max tokens; must truncate or window.
* **Metrics misuse:** only reporting generation accuracy; retrieval must be evaluated separately.
* **Index drift:** embeddings change → FAISS index rebuild required.
* **OOM issues:** dense embeddings on large corpus; mitigate with batching.

# Best Practices

* Log both retrieval and generation configs in MLflow.
* Keep reproducibility: fixed seeds, dataset fingerprints, tokenizer artifacts.
* Unit tests: deterministic retrieval results under fixed embeddings.
* Guardrails in serving: enforce schema, max context length, reject empty queries.
* Baseline → Retriever Comparison → RAG pipeline narrative.

# Reflection & Discussion Prompts

* Why does retrieval improve generation robustness?
* How does index choice (BM25 vs FAISS) affect performance and scalability?
* What risks come from hallucinations in RAG pipelines?

# Next Steps / Advanced Extensions

* Integrate cross-encoder rerankers for retrieval.
* Multi-hop retrieval pipelines.
* Domain adaptation: civic datasets (public documents, meeting notes).
* Lightweight monitoring: retrieval recall drift over time.

# Glossary / Key Terms

RAG, retrieval, generation, ANN, FAISS, BM25, precision\@k, MRR, hallucination.

# Additional

* [[Sentence-Transformers](https://www.sbert.net/)](https://www.sbert.net/)
* [[Haystack](https://docs.haystack.deepset.ai/)](https://docs.haystack.deepset.ai/)
* [[Hugging Face Transformers](https://huggingface.co/docs/transformers)](https://huggingface.co/docs/transformers)
* [[Hugging Face Datasets](https://huggingface.co/datasets)](https://huggingface.co/datasets)
* [[FAISS](https://faiss.ai/)](https://faiss.ai/)
* [[MLflow](https://mlflow.org/)](https://mlflow.org/)
* [[FastAPI](https://fastapi.tiangolo.com/)](https://fastapi.tiangolo.com/)

# Contributors

Author(s): TBD
Reviewer(s): TBD
Maintainer(s): TBD
Date updated: 2025-09-20
Dataset licenses: WikiMovies (open), HotpotQA (CC BY-SA).

# Issues Referenced

Epic: HfLA Text Analysis Tutorials (T0–T14).
This sub-issue: **T11: RAG Basics**.

---

# Resources