Text Analysis Tutorial: Evaluation & Error Analysis

---

# Title & Overview

**Template:** *Evaluation & Error Analysis: An Intermediate, End-to-End Analysis Tutorial*
**Overview (≤2 sentences):** Learners will design evaluation pipelines that go beyond a single accuracy score, incorporating calibration, confusion analysis, and robustness checks. It is intermediate because it teaches structured error analysis, slice-based evaluation, and reproducibility of metrics.

# Purpose

The value-add is building defensible evaluation frameworks that highlight **where and why models fail**. Learners will explore calibration, error taxonomies, slice-based robustness, and logging reproducible evaluation artifacts for applied NLP projects.

# Prerequisites

* Skills: Python, Git, pandas, ML basics (precision/recall/F1).
* NLP: classification, token-level tasks, evaluation metrics (macro/micro F1, AUC).
* Tooling: pandas, scikit-learn, Hugging Face Transformers, torchmetrics/seqeval, MLflow, FastAPI.

# Setup Instructions

* Environment: Conda/Poetry (Python 3.11), deterministic seeds.
* Install: pandas, scikit-learn, torchmetrics, seqeval, Hugging Face Transformers + Datasets, MLflow, FastAPI.
* Datasets:

* **Small:** SST-2 (sentiment classification).
* **Medium:** AG News (topic classification).
* Repo layout:

```
tutorials/t12-eval-error-analysis/
├─ notebooks/
├─ src/
│ ├─ metrics.py
│ ├─ error_analysis.py
│ ├─ calibration.py
│ └─ config.yaml
├─ data/README.md
├─ reports/
└─ tests/
```

# Core Concepts

* **Metrics granularity:** accuracy vs macro/micro F1 vs per-class metrics.
* **Calibration:** Expected Calibration Error (ECE), reliability curves.
* **Error taxonomies:** confusion pairs, failure cases, robustness slices.
* **Slice-based evaluation:** performance by input length, domain, attribute.
* **Reproducibility:** log metrics, seeds, configs, and confusion matrices.

# Step-by-Step Walkthrough

1. **Data intake & splits:** load SST-2/AG News with reproducible splits.
2. **Baseline model predictions:** obtain outputs from Logistic Regression and DistilBERT.
3. **Metric computation:** accuracy, precision, recall, macro/micro F1, AUC (if binary).
4. **Calibration analysis:** compute ECE, plot reliability curves.
5. **Error analysis:**

* Confusion matrices (AG News: misclassifications between Sports/World).
* Hardest examples (highest loss).
* Slice performance (short vs long text, with/without punctuation).
6. **Reporting:** metrics tables, confusion heatmaps, calibration plots, error slices → `reports/t12-eval-error-analysis.md`.
7. *(Optional)* Serve: FastAPI endpoint that accepts predictions + labels and returns full evaluation report.

# Hands-On Exercises

* Compare micro vs macro F1 for imbalanced datasets.
* Ablations: with vs without calibration; with vs without slice metrics.
* Robustness: test on noisy inputs (typos, domain shift).
* Stretch: implement error taxonomy tags (e.g., ambiguity, OOD, adversarial).

# Common Pitfalls & Troubleshooting

* **Metrics misuse:** reporting accuracy only on imbalanced datasets.
* **Over-aggregation:** averaging hides poor class-level performance.
* **Calibration ignored:** models may be accurate but overconfident.
* **Silent errors:** mislabeled data mistaken for model errors.
* **Reproducibility gaps:** missing config/logging prevents repeatable analysis.

# Best Practices

* Always log confusion matrices, calibration plots, and per-slice metrics.
* Use MLflow for versioned metric artifacts.
* Keep Baseline → Error Analysis narrative.
* Unit tests: ensure metrics consistent under toy datasets.
* Guardrails: raise warnings for zero-division in F1.

# Reflection & Discussion Prompts

* Why is macro F1 more informative than accuracy in civic datasets?
* What insights come from calibration beyond precision/recall?
* How would you design slice metrics for fairness-sensitive applications?

# Next Steps / Advanced Extensions

* Automate evaluation in CI pipelines.
* Add robustness checks (counterfactuals, adversarial tests).
* Incorporate fairness metrics (demographic parity, equal opportunity).
* Lightweight monitoring: online calibration drift.

# Glossary / Key Terms

Accuracy, precision, recall, F1, calibration, ECE, reliability curve, confusion matrix, slice metrics.

# Additional Resources

* [[scikit-learn metrics](https://scikit-learn.org/stable/modules/model_evaluation.html)](https://scikit-learn.org/stable/modules/model_evaluation.html)
* [[torchmetrics](https://torchmetrics.readthedocs.io/)](https://torchmetrics.readthedocs.io/)
* [[seqeval](https://github.com/chakki-works/seqeval)](https://github.com/chakki-works/seqeval)
* [[MLflow](https://mlflow.org/)](https://mlflow.org/)
* [[FastAPI](https://fastapi.tiangolo.com/)](https://fastapi.tiangolo.com/)

# Contributors

Author(s): TBD
Reviewer(s): TBD
Maintainer(s): TBD
Date updated: 2025-09-20
Dataset licenses: SST-2 (GLUE), AG News (CC).

# Issues Referenced

Epic: HfLA Text Analysis Tutorials (T0–T14).
This sub-issue: **T12: Evaluation & Error Analysis**.

---