MediaWiki API Project: Geospatial Knowledge Coverage vs. Need

### Overview

Measure **geospatial knowledge coverage vs. need** on Wikipedia by mapping where geotagged articles exist and normalizing by population and economic indicators. Deliver a reproducible dataset and dashboard that highlight under-documented regions and topic gaps.

### Action Items

**If this is the beginning (research & design)**

* Define spatial units (pick 1 to start): **country-level**, **H3 grid** (global at res 5–6), or **city radii** (e.g., 20 km around major cities).
* Select normalization signals: population (required), GDP per capita (optional), poverty rate (optional). Prefer public sources (World Bank API) and/or Wikidata population (P1082) at country/city levels.
* Topic taxonomy: settlements, natural features, culture/heritage, health/education, infrastructure (map via `prop=categories` and/or Wikidata P31).
* Methods choice:

* **Track A (Action API first):** tile world with H3 cells → `list=geosearch` sampling → dedupe pages → `prop=coordinates` and `prop=categories`.
* **Track B (hybrid with Wikidata):** pull items with coordinates (P625), country (P17), population (P1082), instance of (P31) → map to enwiki sitelinks.
* Tooling (choose pairs): `requests` or `httpx`; `pandas` or `polars`; `duckdb` or `sqlite`; **H3** or **geohash**; `altair` or `plotly`. Optional: `geopandas` or `shapely` for country joins.

**If researched and ready (implementation steps)**

1. **Seed spatial units**

* H3 grid: generate global cells at chosen resolution; store centroids + radius.
* Country mode: list ISO countries (csv) for joins; optional Natural Earth boundaries if doing PIP offline.
2. **Collect geotagged pages**

* H3 sampling: for each cell center → `list=geosearch` (`gscoord`, `gsradius`, `gslimit=max`), dedupe by `pageid`.
* Enrich: `prop=coordinates` (type, dim) and `prop=categories&clshow=!hidden` for topic tags.
* Map to QIDs: `prop=pageprops&ppprop=wikibase_item` → Wikidata `wbgetentities` for P31 (topic) and P17/P131 (location) when available.
3. **Normalization data**

* Country-level: population, GDP per capita from World Bank API or Wikidata; city-level: population via Wikidata P1082. Cache to `norm_metrics.parquet`.
4. **Feature engineering**

* Per unit (country/H3/city): `geotagged_article_count`, per-topic counts, **articles per 1M people**, **articles per 1k km²**, and **coverage z-scores** within region/income group.
5. **Analysis**

* Identify under-documented areas (bottom decile per capita coverage), topic skews, and correlations with GDPpc/poverty. Produce hotspot maps and league tables.
6. **Deliver**

* Artifacts: `pages_geo.parquet`, `unit_summary.parquet`, `topic_summary.parquet`, `norm_metrics.parquet`, `metrics.csv`.
* Dashboard: choropleth (country) or H3 hex map, topic mix bars, and “most under-covered” table.
* Methods README with API params, sampling trade-offs, and limitations.
7. **Quality & Ops**

* Caching; retries with exponential backoff; persist `continue` tokens.
* Tests: dedupe integrity, cell-to-page coverage rate, topic mapping precision on a labeled sample.
* Error handling: `try/except` with clear `print()` on file-not-found; **terminate with a trace** on dtype mismatches; log warnings for recoverable issues.
* Optional scheduled monthly refresh (GitHub Actions).

### Resources/Instructions

**MediaWiki / Wikidata docs**

* Action API overview: `API:Action_API`
* Geosearch: `API:Geosearch`
* Coordinates: `API:Coordinates`
* Categories: `API:Categories`
* Pageprops (QID): `API:Pageprops`
* Wikidata API: `wbgetentities` (for P31/P17/P1082)
* World Bank API (optional normalization): Indicators for population and GDP per capita

**Suggested libraries (choose pairs)**

* HTTP: `requests` | `httpx`
* DataFrames: `pandas` | `polars`
* Spatial index: `h3` | `pygeohash`
* Storage: `duckdb` | `sqlite`
* Viz: `altair` | `plotly`
* (Optional) Geospatial: `geopandas` | `shapely`

**Sample queries (copy to notes)**

```
# Geosearch around a cell centroid (10 km radius)
action=query&list=geosearch&gscoord=|&gsradius=10000&gslimit=max

# Get coordinates for a batch of pages
action=query&prop=coordinates&titles=||...&colimit=max

# Categories for topic mapping (exclude hidden)
action=query&prop=categories&clshow=!hidden&titles=

# Page → Wikidata QID
action=query&prop=pageprops&ppprop=wikibase_item&pageids=<PAGEID>

# Wikidata entity lookup (topic, country, population)
action=wbgetentities&ids=<QID>&props=claims|labels|sitelinks
```

**Ethics & reporting**

* Focus on **aggregate** coverage; do not profile individual editors or communities.

* Be explicit: article counts ≠ content quality; coverage ≠ endorsement.

* Document sampling bias (H3 gaps, language bias) and missingness (uncoordinated pages).

* If this issue requires access to 311 data, please answer the following questions:

* Not applicable.
* N/A
* N/A
* N/A

---

## Project Outline (detailed plan for this idea) in details:

<details>

**Research question**
Are locations in low-income or underserved regions **under-documented** on Wikipedia relative to population and geography, and which topics are most lacking?

**Data sources & modules**

* Action API: `list=geosearch`, `prop=coordinates`, `prop=categories`, `prop=pageprops`.
* Wikidata: `wbgetentities` for P31 (instance of), P17 (country), P1082 (population).
* Normalizers: World Bank population/GDPpc (or Wikidata where available).

**Method**

1. Choose spatial unit (country or H3).
2. Collect geotagged pages via geosearch sampling; dedupe by `pageid`.
3. Enrich with categories and Wikidata P31/P17; assign each page to a **topic bucket** and a **spatial unit**.
4. Join population/GDPpc; compute per-capita and per-area article densities and topic shares.
5. Rank units by under-/over-coverage; compute correlations with GDPpc/poverty; visualize gaps.

**Key metrics**

* Articles per 1M people; articles per 1k km².
* Topic mix entropy by region; deficit index vs median of peer income group.
* Top under-covered regions (overall and per topic).

**Deliverables**

* Clean tables (`pages_geo.parquet`, `unit_summary.parquet`, `topic_summary.parquet`, `norm_metrics.parquet`).
* Reproducible notebook + `reports/geo_coverage_vs_need.md`.
* Dashboard: map + tables + topic filters.

**Caveats & limitations**

* Geosearch sampling may miss short-lived or uncategorized pages; language coverage is uneven.
* Country joins via P17/P131 can be incomplete; document fallback logic.
* Per-capita normalization depends on the chosen year/source; pin the version and cite.

**Implementation notes**

* Keys: `(pageid)` for pages; `(h3,res)` or `(iso3)` for units.
* Maintain a query manifest; cache raw responses.
* Provide a small hand-labeled set to validate topic mapping rules; report precision/recall.

</details>