Text Analysis Tutorial: Text Classification (baseline logistic/SVM vs Transformer fine-tune)

# Title & Overview

**Template:** *Text Classification: An Intermediate, End-to-End Analysis Tutorial*
**Overview (≤2 sentences):** Learners will build reproducible text classification pipelines, first with classical models (Logistic Regression/SVM on TF-IDF) and then with transformer fine-tuning (DistilBERT + byte-level BPE). It is intermediate because it stresses defensible baselines, rigorous evaluation (macro-F1, calibration), and slice-level error analysis.

# Purpose

The value-add is learning how to benchmark classical vs neural approaches **fairly** and interpret differences in performance, generalization, and robustness. This teaches reproducibility, error slicing, and serving considerations — skills needed for applied NLP projects, not just leaderboard chasing.

# Prerequisites

* Skills: Python, Git, pandas, ML basics (splits, hyperparameter tuning).
* NLP: TF-IDF, embeddings, byte-level BPE, evaluation metrics (macro-F1).
* Tooling: pandas, scikit-learn, Hugging Face Transformers, MLflow, FastAPI.

# Setup Instructions

* Environment: Conda/Poetry (Python 3.11), deterministic seeds.
* Install: pandas, scikit-learn, Hugging Face Transformers + Datasets, MLflow, FastAPI.
* Datasets:

* **Small:** SST-2 (binary sentiment).
* **Medium:** AG News (4-class topic classification).
* Repo layout:

```
tutorials/t3-text-classification/
├─ notebooks/
├─ src/
│ ├─ models.py
│ ├─ train.py
│ ├─ eval.py
│ └─ config.yaml
├─ data/README.md
├─ reports/
└─ tests/
```

# Core Concepts

* **Baseline-first:** start with TF-IDF + linear models before transformers.
* **Byte-level BPE transformer fine-tuning:** robustness to OOV, emojis, noisy input.
* **Fair comparisons:** identical splits, deterministic seeds, consistent evaluation metrics.
* **Reproducibility:** configs for hyperparameters, tracked artifacts.
* **Error analysis:** confusion matrices, calibration, slice robustness.

# Step-by-Step Walkthrough

1. **Data intake & splits:** stratified reproducible splits, log dataset version.
2. **Classical baseline:**

* TF-IDF (char/word n-grams tuned).
* Classifiers: Logistic Regression (tuned C), Linear SVM.
3. **Transformer baseline:**

* DistilBERT (byte-level BPE).
* Training loop: learning rate warmup, early stopping, fixed seeds.
4. **Evaluation:** macro-F1, accuracy, calibration (ECE), per-slice metrics (short vs long texts).
5. **Error analysis:** confusion pairs, hardest misclassified samples, OOD text checks.
6. **Reporting:** export metrics tables, loss curves, error examples into `reports/t3-text-classification.md`.
7. *(Optional)* Serve: FastAPI endpoint that runs the trained classifier, validates schema, and logs requests.

# Hands-On Exercises

* Compare Logistic Regression vs SVM vs DistilBERT — log metrics table.
* Ablations: TF-IDF n-grams vs char n-grams; transformer freeze ratio (frozen vs full fine-tune).
* Robustness: add spelling noise, rare words; compare TF-IDF vs embeddings.
* Stretch: experiment with smaller transformers (DistilGPT2) for low-resource setups.

# Common Pitfalls & Troubleshooting

* **Unfair comparisons:** transformer gets more epochs/data than classical model.
* **Silent preprocessing drift:** train/test pipelines must be identical.
* **Metrics misuse:** micro-F1 hides poor minority-class performance.
* **OOM on transformers:** use gradient accumulation, truncation.
* **Tokenizer drift:** log tokenizer vocab/merges to avoid silent changes.

# Best Practices

* Always keep Baseline → Improvement → Transformer narrative.
* Log configs, metrics, tokenizer artifacts, and seeds with MLflow.
* Write unit tests for tokenization + vectorization outputs.
* Use schema validation and max-length guardrails in serving.
* Document dataset license and split integrity in `data/README.md`.

# Reflection & Discussion Prompts

* Why do classical models sometimes outperform transformers on small data?
* What do calibration and slice metrics reveal beyond accuracy?
* How would you justify transformer compute cost vs marginal F1 gains in civic-tech settings?

# Next Steps / Advanced Extensions

* Experiment with other small transformers (TinyBERT, MobileBERT).
* Use knowledge distillation: DistilBERT teacher → Logistic Regression student.
* Introduce class imbalance handling (resampling, weighted loss).
* Lightweight monitoring: track prediction drift in serving.

# Glossary / Key Terms

TF-IDF, Logistic Regression, SVM, byte-level BPE, DistilBERT, macro-F1, calibration, ECE, slice metrics.

# Additional Resources

* [[scikit-learn](https://scikit-learn.org/stable/)](https://scikit-learn.org/stable/)
* [[Hugging Face Transformers](https://huggingface.co/docs/transformers)](https://huggingface.co/docs/transformers)
* [[Hugging Face Datasets](https://huggingface.co/datasets)](https://huggingface.co/datasets)
* [[MLflow](https://mlflow.org/)](https://mlflow.org/)
* [[FastAPI](https://fastapi.tiangolo.com/)](https://fastapi.tiangolo.com/)

# Contributors

Author(s): TBD
Reviewer(s): TBD
Maintainer(s): TBD
Date updated: 2025-09-20
Dataset licenses: SST-2 (GLUE), AG News (CC).

# Issues Referenced

Epic: HfLA Text Analysis Tutorials (T0–T14).
This sub-issue: **T3: Text Classification**.

---