Text Analysis Tutorial: Summarization (extractive vs abstractive; length control)

---

# Title & Overview

**Template:** *Summarization: An Intermediate, End-to-End Analysis Tutorial*
**Overview (≤2 sentences):** Learners will build and compare extractive (sentence selection) and abstractive (transformer-generated) summarization systems. It is intermediate because it requires evaluation beyond ROUGE (length control, factual consistency) and error analysis on summary quality.

# Purpose

The value-add is teaching learners how to **design defensible summarization pipelines** by comparing extractive vs abstractive approaches, managing sequence length, and performing structured error analysis. This tutorial stresses reproducibility, metrics, and report generation for summarization tasks.

# Prerequisites

* Skills: Python, Git, pandas, ML basics.
* NLP: tokenization, embeddings, evaluation metrics (ROUGE, BLEU, BERTScore).
* Tooling: pandas, scikit-learn, Hugging Face Transformers, MLflow, FastAPI.

# Setup Instructions

* Environment: Conda/Poetry (Python 3.11), deterministic seeds.
* Install: pandas, scikit-learn, Hugging Face Transformers + Datasets, ROUGE-score, MLflow, FastAPI.
* Datasets:

* **Small:** CNN/DailyMail (summarization, extractive + abstractive).
* **Medium:** XSum (single-sentence abstractive summaries).
* Repo layout:

```
tutorials/t7-summarization/
├─ notebooks/
├─ src/
│ ├─ extractive.py
│ ├─ abstractive.py
│ ├─ eval.py
│ └─ config.yaml
├─ data/README.md
├─ reports/
└─ tests/
```

# Core Concepts

* **Extractive vs abstractive:** selecting vs generating summaries.
* **Length control:** truncation, sentence rank thresholds, max token length.
* **Byte-level BPE:** ensures coverage of rare words in abstractive models.
* **Evaluation:** ROUGE, BLEU, BERTScore; limitations of overlap-based metrics.
* **Error slicing:** summary quality across length buckets, factual consistency checks.

# Step-by-Step Walkthrough

1. **Data intake & splits:** load CNN/DailyMail and XSum; reproducible train/val/test.
2. **Extractive baseline:**

* TextRank or TF-IDF sentence scoring.
* Select top-k sentences by importance.
3. **Abstractive baseline:**

* DistilBART or T5-small with byte-level BPE tokenizer.
* Fine-tuning with teacher forcing; inference with beam search.
4. **Evaluation:** ROUGE-L, BLEU, BERTScore; length compliance metrics.
5. **Error analysis:** over/under-generation, hallucinations, truncation issues.
6. **Reporting:** metrics tables, example summaries, error slices in `reports/t7-summarization.md`.
7. *(Optional)* Serve: FastAPI endpoint with extractive/abstractive options; schema validation, max length guardrail.

# Hands-On Exercises

* Ablations: extractive vs abstractive; greedy vs beam search; different summary lengths.
* Robustness: add noisy sentences; test whether models ignore irrelevant info.
* Slice analysis: compare summary quality for long vs short articles.
* Stretch: constrained decoding (coverage penalty, length penalty tuning).

# Common Pitfalls & Troubleshooting

* **Hallucinations:** abstractive models may invent facts; requires human/error analysis.
* **Metrics misuse:** ROUGE alone ≠ summary quality; complement with BERTScore.
* **Length issues:** truncation may cut key sentences; manage max token length.
* **Extractive bias:** sentence ordering can skew extractive baselines.
* **OOM:** abstractive training with long sequences; mitigate with gradient accumulation.

# Best Practices

* Always compare extractive vs abstractive baselines.
* Track config, seeds, tokenizer artifacts, dataset fingerprints in MLflow.
* Use error slices: factual vs hallucinated summaries, long vs short docs.
* Unit tests: ensure extractive method selects k sentences reproducibly.
* Guardrails: enforce max length and schema validation in serving.

# Reflection & Discussion Prompts

* Why do abstractive models hallucinate more than extractive ones?
* When is extractive summarization preferable in practice?
* How do we balance brevity vs coverage in summary design?

# Next Steps / Advanced Extensions

* Experiment with PEGASUS or Longformer for long-document summarization.
* Evaluate factual consistency with QA-based metrics.
* Domain adaptation: summarization for civic/public documents.
* Lightweight deployment: summarization API with latency monitoring.

# Glossary / Key Terms

Extractive summarization, abstractive summarization, ROUGE, BLEU, BERTScore, hallucination, beam search, length penalty.

# Additional Resources

* [[Hugging Face Transformers](https://huggingface.co/docs/transformers)](https://huggingface.co/docs/transformers)
* [[Hugging Face Datasets](https://huggingface.co/datasets)](https://huggingface.co/datasets)
* [[ROUGE](https://github.com/google-research/google-research/tree/master/rouge)](https://github.com/google-research/google-research/tree/master/rouge)
* [[MLflow](https://mlflow.org/)](https://mlflow.org/)
* [[FastAPI](https://fastapi.tiangolo.com/)](https://fastapi.tiangolo.com/)

# Contributors

Author(s): TBD
Reviewer(s): TBD
Maintainer(s): TBD
Date updated: 2025-09-20
Dataset licenses: CNN/DailyMail (non-commercial), XSum (BBC dataset).

# Issues Referenced

Epic: HfLA Text Analysis Tutorials (T0–T14).
This sub-issue: **T7: Summarization**.

---