Text Analysis Tutorial: Question Answering (extractive)

---

# Title & Overview

**Template:** *Question Answering (Extractive): An Intermediate, End-to-End Analysis Tutorial*
**Overview (≤2 sentences):** Learners will implement extractive QA pipelines, starting with a baseline retriever + reader system (BM25 + lightweight reader) and comparing it to a transformer-based QA model. It is intermediate because it covers retrieval–reading integration, evaluation beyond exact match, and error analysis across question/answer types.

# Purpose

The goal is to teach learners how to **build defensible extractive QA systems**, not just fine-tune a model. They will explore baselines vs transformers, handle retrieval integration, ensure reproducibility, and analyze errors by question type and answer span.

# Prerequisites

* Skills: Python, Git, pandas, ML basics.
* NLP: tokenization, embeddings, retrieval vs reading, evaluation metrics (Exact Match, F1).
* Tooling: pandas, scikit-learn, Hugging Face Transformers, MLflow, FastAPI, Haystack (for retrieval).

# Setup Instructions

* Environment: Conda/Poetry (Python 3.11), deterministic seeds.
* Install: pandas, scikit-learn, Hugging Face Transformers + Datasets, MLflow, FastAPI, Haystack, rank-bm25.
* Datasets:

* **Small:** SQuAD v1.1 (English extractive QA).
* **Medium:** Natural Questions (simplified version from HF Datasets).
* Repo layout:

```
tutorials/t6-question-answering/
├─ notebooks/
├─ src/
│ ├─ retriever.py
│ ├─ reader.py
│ ├─ pipeline.py
│ └─ config.yaml
├─ data/README.md
├─ reports/
└─ tests/
```

# Core Concepts

* **Retrieval + reader pipeline:** separate document retrieval (BM25, ANN) from answer extraction.
* **Byte-level BPE tokenization:** robust handling of unusual tokens in QA contexts.
* **Evaluation metrics:** Exact Match (EM), token-level F1, calibration of confidence scores.
* **Error slicing:** question type (who/what/when/why), answer span length, context length.
* **Reproducibility:** fixed seeds, dataset versions, config-driven runs.

# Step-by-Step Walkthrough

1. **Data intake & splits:** load SQuAD/NQ via HF; split reproducibly.
2. **Baseline pipeline:**

* Retriever: BM25 (rank-bm25).
* Reader: simple linear span predictor (logistic regression on TF-IDF features).
3. **Transformer QA baseline:** DistilBERT (byte-level BPE) fine-tuned on QA with start/end span heads.
4. **Evaluation:** report EM + F1; per-question-type accuracy; calibration (confidence vs correctness).
5. **Error analysis:** hardest spans (longer than max seq-len), ambiguous questions, unanswerable cases.
6. **Reporting:** metrics tables, error categories, example Q/A pairs to `reports/t6-question-answering.md`.
7. *(Optional)* Serve: FastAPI endpoint with retrieval + reader pipeline, schema validation, input length checks.

# Hands-On Exercises

* Ablations: BM25-only vs BM25+reader vs DistilBERT QA.
* Robustness: add noise to contexts (distractor sentences), measure F1 drop.
* Slice evaluation: performance by question type (factoid vs descriptive).
* Stretch: integrate ANN retriever (FAISS) for large corpora.

# Common Pitfalls & Troubleshooting

* **Retriever mismatch:** poor retrieval kills reader performance — must evaluate both separately.
* **Seq-len truncation:** answers may fall outside truncated window; use sliding window inference.
* **Metrics misuse:** EM underestimates near-misses; must pair with F1.
* **Unanswerable questions:** some datasets include them — ensure handling is explicit.
* **Silent drift:** tokenizer vocab changes affect span prediction.

# Best Practices

* Log retriever and reader configs separately.
* Report both EM and F1, per question type.
* Use ablations to understand retriever vs reader contributions.
* Track dataset fingerprints, seeds, and tokenizer artifacts with MLflow.
* Guardrails in serving: max context length, reject empty queries.

# Reflection & Discussion Prompts

* Why does BM25 sometimes outperform transformers on short factoid questions?
* How does context length affect transformer QA performance?
* What’s the trade-off between exact answers and plausible near-misses?

# Next Steps / Advanced Extensions

* Multi-hop QA with chained retrieval.
* Integrate semantic retrievers (Sentence-Transformers, DPR).
* Explore hybrid retrievers (BM25 + dense embeddings).
* Deploy lightweight monitoring: track EM/F1 drift over time.

# Glossary / Key Terms

Retriever, reader, extractive QA, span prediction, Exact Match, F1, sliding window inference, calibration.

# Additional Resources

* [[Hugging Face Transformers](https://huggingface.co/docs/transformers)](https://huggingface.co/docs/transformers)
* [[Hugging Face Datasets](https://huggingface.co/datasets)](https://huggingface.co/datasets)
* [[Haystack](https://docs.haystack.deepset.ai/)](https://docs.haystack.deepset.ai/)
* [[MLflow](https://mlflow.org/)](https://mlflow.org/)
* [[FastAPI](https://fastapi.tiangolo.com/)](https://fastapi.tiangolo.com/)

# Contributors

Author(s): TBD
Reviewer(s): TBD
Maintainer(s): TBD
Date updated: 2025-09-20
Dataset licenses: SQuAD (Stanford, CC BY-SA), Natural Questions (Google, CC).

# Issues Referenced

Epic: HfLA Text Analysis Tutorials (T0–T14).
This sub-issue: **T6: Question Answering (Extractive)**.

---