MediaWiki API Project: Category Under/Over-Coverage

### Overview

Quantify **category-level under/over-coverage** on Wikipedia by crawling sensitive topical categories (and their subcategories) and measuring article counts, content depth, references, and quality signals—then ranking categories against their peers to surface gaps.

### Action Items

**If this is the beginning (research & design)**

* Define seed categories (e.g., human rights, policing, immigration, public health, Indigenous issues). Set a max traversal depth (start with 1–2).
* Decide metrics: pages per category, median article length, references/article, % with infobox, interlanguage links count, PageAssessments quality/importance (if enabled), and **additions per month** (from category-membership timestamps).
* Methods plan: breadth-first search of subcategories via `list=categorymembers`; fetch page features via `prop=revisions` (size), `prop=revisions&rvprop=content` (for `` and infobox detection), `prop=langlinks` (coverage across languages), `prop=pageassessments` (quality, if available).
* Tooling (pick pairs): `requests` or `httpx`; `pandas` or `polars`; storage in `duckdb` or `sqlite`; visualization in `Altair` or `Plotly`.

**If researched and ready (implementation steps)**

1. **Seed & crawl**

* For each top-level category: `list=categorymembers&cmtype=subcat|page&cmlimit=max&cmnamespace=0|14`.
* BFS to a fixed depth; persist `(category, pageid, title, member_timestamp)`. De-duplicate pages across subcats.
2. **Feature extraction**

* Size & timestamps: `prop=revisions&rvprop=size|timestamp&rvlimit=1&rvdir=older/newer` as needed.
* References & infobox: fetch wikitext for sampled/latest revids and count `

# Fetch content for counting and {{Infobox
action=query&prop=revisions&revids=&rvslots=main&rvprop=content

# Interlanguage links count
action=query&prop=langlinks&lllimit=max&titles=

# Page assessments (if available)
action=query&prop=pageassessments&titles=<TITLE>
```

**Ethics & caveats**

* Category systems are noisy; membership can be incomplete or inconsistent. Document traversal depth and exclusions (hidden cats if you choose).

* Ref/infobox counts are proxies for depth, not quality. Treat PageAssessments as optional (not all wikis have it).

* Aggregate reporting only; no editor-level profiling.

* If this issue requires access to 311 data, please answer the following questions:

* Not applicable.
* N/A
* N/A
* N/A

---

## Project Outline (detailed plan for this idea) in details:

<details>

**Research question**
Which sensitive topical categories are **thin** (few pages, short articles, sparse references, low interlanguage coverage) relative to peer categories, and how are they changing month to month?

**Data sources & modules**

* `list=categorymembers` for pages/subcats (+ timestamps).
* `prop=revisions` (size/content).
* `prop=langlinks` (language breadth).
* `prop=pageassessments` (quality/importance, when available).

**Method**

1. BFS the category graph to a fixed depth, collecting member pages and timestamps.
2. For each page, compute size, ref count, infobox presence, langlinks count, and optional quality labels.
3. Aggregate per category; build **Under-Coverage Index** and rank.
4. Produce monthly addition trends and highlight categories in the bottom decile on ≥2 core metrics.

**Key metrics**

* `page_count`, `median_size`, `median_refs_per_article`, `%_infobox`, `median_langlinks`, quality mix.
* Monthly additions; 12-mo Δ in page\_count and size.
* Under-Coverage Index and bottom-decile flags.

**Deliverables**

* Clean tables (`category_members.parquet`, `page_features.parquet`, `category_monthly.parquet`, `category_metrics.csv`).
* Notebook + `reports/category_under_over_coverage.md`.
* Streamlit/Altair dashboard (ranked table, sparklines, metric drill-downs).

**Limitations**

* Category membership timestamps can reflect when categorization occurred, not page creation; call that out.
* Regex heuristics for refs/infobox may miss template edge cases—validate on a labeled subset and report precision.
* Interlanguage links vary by page age; consider age-adjusted comparisons if needed.

</details>