MediaWiki API Project: Gendered/Respectful Language Drift (think pronouns)

### Overview

Track **gendered/respectful language drift** on Wikipedia by measuring how usage of a curated term list (deprecated → preferred) changes over time on policy-relevant pages. Deliver a reproducible dataset, statistical deltas, and a small dashboard highlighting rise/fall, first-adoption dates, and context snippets.

### Action Items

**If this is the beginning (research & design)**

* Define scope: English Wikipedia, Namespace 0; 100–300 pages across immigration, policing, elections, public health, and human rights (`seed_pages.csv` with `title,pageid,qid`).
* Build `terms.csv`: columns `deprecated,preferred,aliases,regex,notes` (e.g., `whitelist→allowlist`, `master/slave→primary/replica`, `illegal immigrant→undocumented immigrant`). Include canonicalization rules (hyphen/space variants).
* Choose windows: monthly sampling from 2016 → present (adjustable).
* Metrics: term frequency per page/month, log-odds (with informative Dirichlet prior) past vs recent, first-seen date for preferred terms, disappearance of deprecated terms, context windows (±25 tokens).
* Tooling (pick one per pair): `requests` or `httpx`; `pandas` or `polars`; `mwparserfromhell` or `wikitextparser`; storage `duckdb` or `sqlite`; viz `altair` or `plotly`.

**If researched and ready (implementation steps)**

1. **Seed & resolve**

* Resolve `pageid` via `action=query&titles=`; store redirects.
2. **Revision sampling**

* For each page/month, fetch the **oldest** revision in the month (creation if earlier) and the **latest** revision (`prop=revisions&rvprop=ids|timestamp&rvlimit=max` to list; then pull `rvslots=main&rvprop=content` for targeted revids).
3. **Text extraction & normalization**

* Convert wikitext → plain text; drop templates, infoboxes, references, code/pre blocks, and quotes where feasible; lowercase + Unicode normalize; apply alias/regex normalization from `terms.csv`.
4. **Counting & context**

* Count deprecated/preferred terms; capture context windows (±25 tokens) for QA; flag code/citation hits for exclusion.
5. **Analysis**

* Compute per-term Δfrequency, Δshare (preferred / (preferred+deprecated)), and log-odds change between early (e.g., 2016–2018) and recent (last 12 months). Identify **newly adopted** (now>0, then=0) and **disappeared** (then>0, now=0) terms.
6. **Deliver**

* Artifacts: `page_month_samples.parquet`, `term_counts.parquet`, `term_context.parquet`, `metrics.csv`.
* Dashboard: rising/falling terms, adoption timelines, page-level diff view with context snippets.
* Methods README (API params, sampling rationale, exclusions, limitations).
7. **Quality & Ops**

* Caching + retries with exponential backoff; honor `maxlag`; persist `continue` tokens.
* Tests: regex correctness, sampler month coverage, exclusion filters, deterministic counts on a fixed snapshot.
* Optional monthly refresh via GitHub Actions.

### Resources/Instructions

**API docs to pin in repo**

```
MediaWiki Action API: https://www.mediawiki.org/wiki/API:Action_API
Query & continuation: https://www.mediawiki.org/wiki/API:Query
Revisions (timestamps, content): https://www.mediawiki.org/wiki/API:Revisions
Pages → pageids: https://www.mediawiki.org/wiki/API:Query#By_page_title
```

**Suggested libraries (choose pairs)**

* HTTP: `requests` | `httpx`
* DataFrames: `pandas` | `polars`
* Parsing: `mwparserfromhell` | `wikitextparser`
* Storage: `duckdb` | `sqlite`
* Viz: `altair` | `plotly`

**Sample queries to copy into notes**

```
# Resolve pageids from titles
action=query&titles=<TITLE>

# List revisions to find oldest/newest in a month (use continuation)
action=query&prop=revisions&rvprop=ids|timestamp&rvlimit=max&titles=<TITLE>

# Fetch content for a specific revision id
action=query&prop=revisions&revids=<REVID>&rvslots=main&rvprop=content
```

**Data handling & ethics**

* Report **aggregates**; avoid editor-level profiling.

* Exclude quotes, code, and citation/reference sections to reduce false positives.

* Terms can be context-sensitive; keep a manual review sheet for ambiguous hits.

* Error handling: wrap I/O in `try/except` with clear `print()` messages for file-not-found; **terminate with a trace** on dtype mismatches; log warnings for recoverable issues.

* If this issue requires access to 311 data, please answer the following questions:

* Not applicable.
* N/A
* N/A
* N/A

---

## Project Outline (detailed plan for this idea) in details:

<details>

**Research question**
Are deprecated phrasings declining and preferred, respectful alternatives rising on sensitive Wikipedia pages, and when did adoption inflect?

**Data sources & modules**

* `prop=revisions` for listing revisions and fetching monthly oldest/newest content.
* `action=query` for title→pageid resolution.

**Method**

1. Curate `terms.csv` with deprecated→preferred mappings and regex/alias variants; add ignore rules (citations, code, quotes).
2. Monthly sample each page: pull oldest and newest revisions; extract and normalize text; count term occurrences and capture contexts.
3. Aggregate per term and per page: frequency, share, log-odds change, first/last seen dates; compute adoption timelines.
4. Visualize rise/fall and adoption curves; surface pages with largest improvements or regressions.

**Key metrics**

* Δfrequency and Δshare preferred vs deprecated.
* Log-odds change (past vs recent).
* Time-to-adoption for preferred term; **disappearance rate** for deprecated term.
* Pages with largest positive/negative shifts.

**Deliverables**

* Clean tables (`term_counts.parquet`, `term_context.parquet`, `term_metrics.parquet`).
* Reproducible notebook + `reports/language_drift.md`.
* Streamlit/Altair dashboard (rising/falling, timelines, per-page diff with snippets).

**Caveats & limitations**

* Sampling misses intra-month churn; choose monthly/quarterly depending on load.
* Some terms have legitimate historical usage in quotes—exclusion rules mitigate but won’t be perfect.
* Wikitext parsing can miss template-embedded text; document residual noise.

**Implementation notes**

* Keys: `(pageid, revid)` for text pulls; `(pageid, month)` for samples.
* Store a query manifest (params + timestamps + continuation cursors).
* Provide a reviewer checklist to confirm mappings and spot-check context windows.

</details>