MediaWiki API Project: Cross-Language Topic Presence for Crisis Terms

### Overview

Measure **cross-language topic presence for crisis terms** (disasters, epidemics, refugee emergencies) by checking if key pages exist across Wikipedias and how fresh they are vs. a reference language. Deliver a reproducible dataset and a dashboard with **coverage**, **time-to-presence**, and **update lag** per topic × language.

### Action Items

**If this is the beginning (research & design)**

* Define the topic list: \~50 English pages for recent crises (disasters, outbreaks, refugee events). Store in `topics.csv` with `title, qid, crisis_type, start_date` (start\_date optional; use Wikidata P580 when available).
* Choose language cohort: top 20 by size **plus** 10 low-resource wikis; or “all languages returned by each topic’s langlinks.”
* Metrics & windows: coverage (exists Y/N), time-to-presence (first creation vs. reference language), **update lag** (latest edit delta vs. reference), optional pageviews for demand context.
* Methods: MediaWiki Action API (`prop=langlinks`, `prop=revisions`, `list=sitematrix`), optional Wikidata (`wbgetentities` for QIDs, P580), optional Pageviews REST.
* Stack (pick pairs; keep consistent): `requests` **or** `httpx`; `pandas` **or** `polars`; `duckdb` **or** `sqlite`; `Altair` **or** `Plotly`.

**If researched and ready (implementation steps)**

1. **Seed topics**

* Build `topics.csv` with English titles and QIDs; include `start_date` where known (manual or Wikidata P580).
2. **Enumerate languages**

* Pull `list=sitematrix` for valid wikis; for each English title get `prop=langlinks` to map to localized titles (store `(lang, title_local)`).
3. **Fetch revision metadata**

* For English and each localized title: `prop=revisions` to get **first** and **latest** timestamps (creation + freshness). Handle redirects.
4. **Compute metrics**

* Coverage rate per language; **time-to-presence** = first\_nonEN − first\_EN (days); **update lag** = latest\_EN − latest\_nonEN (days).
* Optional: compare vs median of top-N languages, not just EN.
5. **Deliver**

* Artifacts: `lang_presence.parquet`, `first_latest.parquet`, `lags.parquet`, `metrics.csv`.
* Dashboard: heatmap of lags (languages × topics), coverage bars, language and topic leaderboards.
* Methods README with exact API params and caveats.
6. **Quality & Ops**

* Caching + retries with exponential backoff; honor `maxlag`; persist `continue` tokens.
* Unit tests: timestamp math, langlink joins, redirect handling.
* Error handling: wrap I/O in `try/except` with informative `print()` on file-not-found; **raise with trace** on dtype mismatches; warn on recoverables.
* Optional: monthly GH Action to refresh a subset.

### Resources/Instructions

**Docs to pin in the repo**

* MediaWiki Action API (overview) — `API:Action_API`
* Interlanguage links — `API:Langlinks`
* Revisions (timestamps, size) — `API:Revisions`
* Site matrix (language list) — `API:Sitematrix`
* Wikidata API — `wbgetentities` (use P580 “start time” when available)
* Pageviews (REST, optional) — AQS Pageviews per-article

**Suggested libraries (choose pairs)**

* HTTP: `requests` | `httpx`
* Frames: `pandas` | `polars`
* Storage: `duckdb` | `sqlite`
* Viz: `Altair` | `Plotly`

**Sample queries to copy**

```
# Interlanguage links for an English topic
action=query&prop=langlinks&titles=&lllimit=max

# Latest revision timestamp for a localized title
action=query&prop=revisions&rvprop=timestamp|size&rvlimit=1&titles=

# First revision timestamp (creation)
action=query&prop=revisions&rvprop=timestamp&rvlimit=1&rvdir=newer&titles=

# Language list
action=sitematrix&formatversion=2

# Wikidata entity (optional: start date P580)
action=wbgetentities&ids=<QID>&props=claims|labels
```

**Ethics & caveats**

* Presence/freshness are **proxies**, not content quality.

* Names can diverge across languages (merges/splits); record redirects and missing pages.

* Aggregate reporting only; no editor-level profiling.

* If this issue requires access to 311 data, please answer:

* Not applicable.
* N/A
* N/A
* N/A

---

## Project Outline (detailed plan for this idea) in details:

<details>

**Research question**
For crisis topics, do smaller Wikipedias **create** pages later and **update** them less frequently than larger ones, and which languages or topics show the largest gaps?

**Data sources & modules**

* `prop=langlinks` (cross-language titles), `list=sitematrix` (wiki list), `prop=revisions` (first/latest).
* Optional: Wikidata P580 (start time), REST Pageviews for demand.

**Method**

1. Curate `topics.csv` (title, qid, crisis\_type, start\_date).
2. For each topic, get localized titles via `langlinks`; validate against `sitematrix`.
3. Pull **first** and **latest** timestamps per language title; normalize to UTC.
4. Compute coverage, time-to-presence, and update lags; summarize by language family/region and crisis\_type.
5. Visualize lag heatmaps and coverage distributions; highlight outliers.

**Key metrics**

* Coverage rate per language (% of topics present).
* Median **time-to-presence** (days).
* Median **update lag** (days) and % topics with lag > thresholds (e.g., 30/90).
* (Optional) Demand-freshness mismatch: correlate lag with pageviews.

**Deliverables**

* Clean tables (`lang_presence.parquet`, `first_latest.parquet`, `lags.parquet`, `metrics.csv`).
* Notebook + `reports/crisis_cross_language_presence.md`.
* Streamlit/Altair dashboard (lag heatmap, coverage bars, rankings).

**Limitations**

* Some crises are tracked as sections within broader pages; treat as “missing” unless a dedicated page exists (document rule).
* Title drift and redirects require careful normalization.
* Timestamp proxies do not capture semantic parity or completeness.

</details>