MediaWiki API Project: Topic-Network Bias and Agenda Setting

### Overview

Map **topic-network bias and agenda setting** on Wikipedia by building page→page link graphs for sensitive topics and measuring concentration, centrality, and community structure. Output a reproducible dataset and dashboard that surface echo-chamber patterns and cross-topic link gaps.

### Action Items

**If this is the beginning (research & design)**

* Define scope: 100–300 English Wikipedia seed pages across elections, policing, migration, public health, human rights, climate (`seed_pages.csv` with `title,pageid,topic`).
* Crawl plan: 1-hop (seed → outlinks) with optional 2-hop expansion; restrict to Namespace 0.
* Metrics: in/out-degree, PageRank, betweenness, **assortativity by topic**, community modularity (Louvain/Leiden), and **source concentration** (HHI of outlinks).
* Tooling (pick pairs and keep consistent): `requests` or `httpx`; `pandas` or `polars`; `networkx` or `igraph`; `duckdb` or `sqlite`; `altair` or `plotly`.
* Governance: log queries/continuations; handle redirects; document namespace filters and exclusions.

**If researched and ready (implementation steps)**

1. **Seed & resolve**

* Resolve `pageid` for titles; normalize canonical titles; capture redirects.
2. **Collect links**

* For each seed: `prop=links&plnamespace=0&pllimit=max` (paginate via `continue`) → outgoing edges.
* Optional incoming context: `list=backlinks&blnamespace=0&bllimit=max` for seed pages.
3. **Topic tags**

* Attach `topic` from the seed list; for neighbors, infer topic via seed that linked them and/or categories (`prop=categories&clshow=!hidden`) with a small mapping ruleset.
4. **Graph build & features**

* Build directed graph `G`; compute degree, PageRank, betweenness, HHI of outlinks, Louvain/Leiden communities, and **assortativity by topic**.
5. **Analysis**

* Identify high-centrality pages, tightly clustered communities (high modularity), and **cross-topic link scarcity**; rank seeds by outlink diversity and dependency on a few hubs.
6. **Deliver**

* Artifacts: `edges.parquet`, `nodes.parquet`, `node_metrics.parquet`, `community_assignments.parquet`, `metrics.csv`.
* Dashboard: centrality leaderboards, community map, topic-assortativity and outlink diversity charts.
7. **Quality & Ops**

* Caching, retries with exponential backoff; honor `maxlag`; store raw JSON & query manifests.
* Tests: continuation correctness, namespace filter, redirect normalization, degree/PageRank sanity checks.
* Error handling: wrap I/O in `try/except` with clear `print()` on file-not-found; **raise with trace** on dtype mismatches; warn on recoverables.

### Resources/Instructions

**API docs to pin in repo**

* Action API overview: `API:Action_API`
* Links (outgoing): `API:Links`
* Backlinks (incoming): `API:Backlinks`
* Categories: `API:Categories`
* Query & continuation etiquette: `API:Query`

**Suggested libraries (choose pairs)**

* HTTP: `requests` | `httpx`
* DataFrames: `pandas` | `polars`
* Graph: `networkx` | `igraph` (communities: `python-louvain` | `leidenalg`)
* Storage: `duckdb` | `sqlite`
* Viz: `altair` | `plotly`

**Sample queries (copy to notes)**

```
# Resolve pageid from title
action=query&titles=

# Outgoing links (Namespace 0)
action=query&prop=links&plnamespace=0&pllimit=max&titles=<TITLE>

# Incoming links (optional)
action=query&list=backlinks&blnamespace=0&bllimit=max&bltitle=<TITLE>

# Categories for topic mapping (exclude hidden)
action=query&prop=categories&clshow=!hidden&titles=<TITLE>
```

**Data handling & ethics**

* Aggregate reporting; avoid editor-level attributions.

* Category systems are noisy; document mapping rules and report coverage/precision.

* Note that links indicate structure, not endorsement.

* If this issue requires access to 311 data, please answer the following questions:

* Not applicable.
* N/A
* N/A
* N/A

---

## Project Outline (detailed plan for this idea) in details:

<details>

**Research question**
Do sensitive-topic pages form insular link communities (echo chambers), and which pages act as cross-topic bridges or central hubs?

**Data sources & modules**

* `prop=links` (outlinks), `list=backlinks` (optional inlinks), `prop=categories` (topic hints), `action=query` (title→pageid).

**Method**

1. Build the seed cohort; resolve canonical titles/pageids.
2. Fetch outlinks for each seed; optionally fetch backlinks for seeds.
3. Normalize nodes; assign topic tags to seeds; infer for neighbors via category mapping.
4. Construct `G` and compute: in/out-degree, PageRank, betweenness, Louvain/Leiden communities, topic assortativity, and HHI of outlinks.
5. Rank seeds by: (a) dependency on a few hubs (high HHI), (b) bridge potential (betweenness), (c) echo-chamber tendency (high assortativity + low cross-topic edges).

**Key metrics**

* Centrality: PageRank, betweenness; degree.
* Diversity: outlink HHI; cross-topic edge share.
* Structure: modularity, topic assortativity coefficient.
* Gaps: pages with low cross-topic links despite high centrality.

**Deliverables**

* Clean tables (`edges.parquet`, `nodes.parquet`, `node_metrics.parquet`, `community_assignments.parquet`).
* Notebook + `reports/topic_network_bias.md`.
* Streamlit/Altair dashboard: community map, centrality tables, diversity charts.

**Limitations**

* Topic inference for neighbors is heuristic; validate with a labeled subset and report precision/recall.
* Backlink collection at scale can be heavy—limit to seeds or sample.
* Link structure doesn’t capture sentiment or citation quality; interpret cautiously.

**Implementation notes**

* Keys: node=`normalized_title` (also keep `pageid`), edge=(source\_title, target\_title).
* Persist query manifests and continuation cursors; cache raw responses.
* Add unit tests for redirect normalization and degree totals (∑in=∑out on the induced subgraph).

</details>