MediaWiki API Project: Source-Type Composition and "Agenda Setting" Sources

### Overview

Audit **source-type composition** and potential agenda setting on Wikipedia by building link graphs for sensitive topics and measuring whether pages disproportionately link to narrow sets of sources (e.g., state media, tabloids, academic journals, reputable newspapers). Deliver a reproducible dataset and dashboard that show concentration, source diversity, and cross-topic differences.

### Action Items

**If this is the beginning (research & design)**

* Define scope: 100–300 English Wikipedia pages across elections, policing, migration, public health, human rights, and climate disinformation (`seed_pages.csv`).
* Build a **source taxonomy**: `source_rules.yaml` mapping URL patterns → classes (e.g., peer-reviewed journal, mainstream news, state media, think tank, government site, company site, tabloid, blog). Include overrides and known aliases/redirects.
* Metrics & windows: current-state snapshot plus a 3–5 year trend (annual). Metrics include: unique sources per page, **Herfindahl–Hirschman Index (HHI)** of sources, share by class, top domains, and change over time.
* Methods:

* Extract references/links from article content via revisions API (`prop=revisions&rvslots=main&rvprop=content`) for selected waypoints (now and past years).
* Parse citations and external links (template fields like `|url=` and bare links), normalize URLs, and classify to taxonomy.
* Build page→source bipartite graph and compute per-page concentration and per-topic distributions.
* Tooling (choose pairs and keep consistent): `requests` or `httpx`; `pandas` or `polars`; `mwparserfromhell` or `wikitextparser`; storage `duckdb` or `sqlite`; viz `altair` or `plotly`; graph `networkx` or `igraph`.
* Ethics: aggregate reporting; avoid naming individual editors; clarify that links ≠ endorsement and many links are citations.

**If researched and ready (implementation steps)**

1. **Seed & resolve**

* Ingest `seed_pages.csv`; resolve `pageid` and record redirects.
2. **Timepoints & pulls**

* For each page, fetch content for `t0` (e.g., Jan 1 three years ago), `t1` (Jan 1 two years ago), `t2` (Jan 1 last year), and `t_now`. Use `prop=revisions` to list revids around those dates and then fetch content for selected `revids`.
3. **Extract & normalize**

* From wikitext, extract: (a) citation templates’ `|url=` fields, (b) external links `[http(s)://...]`, (c) archive URLs → expand to original when present. Normalize to registrable domain + path stem; drop tracking params; handle `doi:` and `pmid:` separately.
4. **Classify sources**

* Apply `source_rules.yaml` (domain regex + path hints) to map each URL to a class; add a small manual override list and an “unknown” bucket.
5. **Graph & metrics**

* Build bipartite graph (page ↔ source domain). Compute per-page: unique source count, HHI, top source share, class shares. Aggregate per topic and over time.
6. **Deliver**

* Artifacts: `links_raw.parquet`, `sources_classified.parquet`, `page_metrics.parquet`, `topic_yearly.parquet`, `graph_edgelist.parquet`, `metrics.csv`.
* Dashboard: per-topic class shares, HHI distributions, top domains table, change-over-time charts.
* Methods README with parsing heuristics, taxonomy, and known edge cases (templates, archives).
7. **Quality & Ops**

* Caching and retries; persist raw JSON; version the taxonomy rules.
* Tests: URL parser precision, classifier coverage, HHI sanity checks.
* Optional: scheduled yearly refresh; diff reports.

### Resources/Instructions

**API docs to pin in repo**

* Action API overview: `API:Action_API`
* Revisions (timestamps, content): `API:Revisions`
* Query continuation & etiquette: `API:Query`
* (Optional) Exturlusage (to spot-check present-day links to a given domain): `API:Exturlusage`

**Suggested libraries (choose pairs)**

* HTTP: `requests` | `httpx`
* DataFrames: `pandas` | `polars`
* Parsing: `mwparserfromhell` | `wikitextparser`
* Storage: `duckdb` | `sqlite`
* Graph: `networkx` | `igraph`
* Viz: `altair` | `plotly`

**Sample queries**

```
# Revisions near a given date (to pick a waypoint revid)
action=query&prop=revisions&rvprop=ids|timestamp&rvlimit=max&rvstart=2022-01-02T00:00:00Z&rvend=2021-12-31T00:00:00Z&titles=

# Fetch content for a specific revision id
action=query&prop=revisions&revids=<REVID>&rvslots=main&rvprop=content

# Present-day pages that link a domain (spot check)
action=query&list=exturlusage&euquery=example.com&eulimit=max&eunamespace=0
```

**Data handling & ethics**

* Report at page/topic aggregates; avoid editor-level commentary.

* Note that many links are citations; classify “archive.org” by its original URL when available.

* Keep an “unknown/unclassified” class; document coverage %.

* Error handling: `try/except` with helpful prints for file-not-found; **terminate with a trace** on dtype mismatches; warn on partial parsing.

* If this issue requires access to 311 data, please answer the following questions:

* Not applicable.
* N/A
* N/A
* N/A

---

## Project Outline (detailed plan for this idea) in details:

<details>

**Research question**
Do sensitive-topic pages rely on a narrow set of source types (low diversity/high concentration), and how has the mix shifted over the last 3–5 years?

**Data sources & modules**

* `prop=revisions` (content) at yearly waypoints.
* `list=exturlusage` for current-state validation.
* Local taxonomy (`source_rules.yaml`) and override list.

**Method**

1. Define page cohort and taxonomy rules.
2. Pull wikitext for 3–4 timepoints per page; extract and normalize URLs (expand archived links to originals).
3. Classify each URL into a source class; compute page-level and topic-level metrics; construct a bipartite graph.
4. Analyze diversity (HHI), top-source share, and class shares over time; identify pages/topics with unusually high concentration.

**Key metrics**

* Unique sources per page; HHI; top-source share.
* Class distribution (% journals, % mainstream news, % state media, etc.).
* Change per year; pages entering/leaving high-concentration status.
* Coverage of classification rules (% URLs classified).

**Deliverables**

* Clean tables (`links_raw.parquet`, `sources_classified.parquet`, `page_metrics.parquet`, `topic_yearly.parquet`, `graph_edgelist.parquet`).
* Notebook + `reports/source_type_audit.md`.
* Streamlit/Altair dashboard with diversity plots and top domains.

**Caveats & limitations**

* Citation templates vary by article; some links are nested or parameterized.
* Archive links and URL shorteners require expansion; some originals are unreachable.
* Classification is heuristic—maintain review samples and report precision/coverage.

**Implementation notes**

* Keys: `(pageid, revid, url_norm)`; normalize domains (registrable) and paths (trim UTM).
* Persist a query manifest and rule versions with artifacts.
* Add small labeled sets to validate parsing and classification; publish confusion examples.

</details>