Text Analysis Tutorial: Text Cleaning & Tokenization: An Intermediate, End-to-End Analysis Tutorial

---

# Title & Overview

**Template:** *Text Cleaning & Tokenization: An Intermediate, End-to-End Analysis Tutorial*
**Overview (≤2 sentences):** Learners will implement a reproducible text cleaning and tokenization pipeline using regex-based normalization and subword tokenization with **byte-level BPE**. This is intermediate because it emphasizes Unicode policies, OOV handling, measurable trade-offs, and reproducible configurations beyond simple whitespace splitting.

# Purpose

To move from ad-hoc tokenization toward a **defensible, configurable pipeline** that works for both classical features (TF-IDF) and transformer fine-tunes. Learners will quantify how cleaning choices and subword tokenization strategies impact metrics, reproducibility, and robustness.

# Prerequisites

* Skills: Python, Git, virtual envs; basic NumPy/pandas; ML basics (splits, regularization).
* NLP: tokenization (word vs subword), embeddings vs TF-IDF, macro-F1 vs accuracy.
* Tooling: **pandas**, **scikit-learn**, **spaCy**, **Hugging Face Transformers**, **MLflow**, **FastAPI**.

# Setup Instructions

* Environment: Conda/Poetry (Python 3.11), deterministic seeds, `.env` for secrets/paths.
* Install: pandas, scikit-learn, spaCy, Hugging Face Transformers + Datasets, MLflow, FastAPI.
* Datasets:

* **Small:** SST-2 (sentiment classification).
* **Medium:** AG News (topic classification).
* Repo layout:

```
tutorials/t1-tokenization/
├─ notebooks/
├─ src/
│ ├─ cleaning.py
│ ├─ tokenizers.py
│ ├─ utils.py
│ └─ config.yaml
├─ data/README.md
├─ reports/
└─ tests/
```

# Core Concepts

* **Byte-level BPE:** avoids OOV by working on raw bytes, preserves emoji/unicode.
* **Cleaning policy:** Unicode normalization (NFC vs NFKC), case folding, URL/userhandle masking, digit masking.
* **Baseline-first:** classical vs small transformer pipeline.
* **Reproducibility:** config-driven cleaning/tokenization policies, seed fixing, dataset versioning.
* **Separation:** modeling vs guardrails (input validation, schema, length checks).

# Step-by-Step Walkthrough

1. **Data intake & splits:** load datasets via HF Datasets, reproducible splits, save metadata.
2. **Cleaning policies:**

* Minimal: NFC, strip control chars, mask URLs.
* Standard: NFKC, lowercase, mask digits, collapse whitespace.
* Aggressive: standard + punctuation spacing, emoji demojize.
Policies tracked in config.
3. **Tokenization:**

* Rule-based (spaCy) for comparison.
* **Byte-level BPE** (GPT-2 tokenizer via HF) as main path.
4. **Baselines:**

* Classical: TF-IDF (char/word n-grams) + Logistic Regression.
* Neural: DistilBERT with byte-level BPE tokenizer, fine-tune with warmup + early stopping.
5. **Evaluation:** macro-F1, accuracy; calibration (ECE); slice metrics (length quartiles, URL/emoji presence).
6. **Error analysis:** confusion pairs, hardest examples, tokenization diffs, OOD checks.
7. **Reporting:** generate `reports/t1-tokenization.md` with nbconvert.
8. *(Optional)* Serve: FastAPI endpoint with schema validation and cleaning policy enforced.

# Hands-On Exercises

* Ablations: minimal vs standard vs aggressive cleaning; seq-len 128 vs 256 vs 512; TF-IDF n-gram ranges.
* Robustness: inject typos/emoji/URLs; measure macro-F1 drift.
* Slice evaluation: compare performance on emoji-heavy vs numeric-heavy samples.
* Stretch: hybrid features — concatenate TF-IDF char-ngrams with transformer embeddings.

# Common Pitfalls & Troubleshooting

* Tokenizer–model mismatch: must pair byte-level BPE tokenizer with compatible transformer (e.g., GPT-2/DistilGPT2).
* Over-cleaning: case folding or deaccenting may collapse useful signals.
* Metrics misuse: prefer macro-F1 in imbalanced cases.
* OOM: long seqs; mitigate with truncation and gradient accumulation.
* I/O errors: wrap loaders in `try/except`; fail fast on dtype mismatches.

# Best Practices

* Log config, seeds, dataset fingerprints, git commit hash, and tokenizer files.
* Always keep Baseline → Improvement narrative.
* Unit tests: deterministic token boundaries, TF-IDF vocab size, subword consistency.
* Separate cleaning/tokenization from model training for clarity.
* Guardrails: enforce max input length and validate schema at serving.

# Reflection & Discussion Prompts

* Why does byte-level BPE improve robustness on noisy text (emoji, rare chars)?
* What trade-offs come from Unicode normalization (NFC vs NFKC)?
* How does case folding impact sentiment vs entity-heavy tasks?

# Next Steps / Advanced Extensions

* Train a custom byte-level BPE tokenizer on domain data (SentencePiece).
* Knowledge distillation: transformer → linear TF-IDF model.
* Semantic search: FAISS over embeddings with byte-BPE tokenization.
* Lightweight fairness check: compare performance across slices with/without emoji.

# Glossary / Key Terms

Byte-level BPE, OOV, NFC/NFKC, macro-F1, calibration, ECE, slice, tokenizer–model compatibility.

# Additional Resources

* [[scikit-learn](https://scikit-learn.org/stable/)](https://scikit-learn.org/stable/)
* [[spaCy](https://spacy.io/)](https://spacy.io/)
* [[Hugging Face Transformers](https://huggingface.co/docs/transformers)](https://huggingface.co/docs/transformers)
* [[Hugging Face Datasets](https://huggingface.co/datasets)](https://huggingface.co/datasets)
* [[MLflow](https://mlflow.org/)](https://mlflow.org/)
* [[FastAPI](https://fastapi.tiangolo.com/)](https://fastapi.tiangolo.com/)

# Contributors

Author(s): TBD
Reviewer(s): TBD
Maintainer(s): TBD
Date updated: 2025-09-20
Dataset licenses: SST-2 (GLUE), AG News (CC).

# Issues Referenced

Epic: HfLA Text Analysis Tutorials (T0–T14).
This sub-issue: **T1: Text Cleaning & Tokenization**.

---