MediaWiki API Project: Newcomer Editors and Gatekeeping

### Overview

Measure **newcomer burden and gatekeeping** on sensitive Wikipedia pages by comparing revert rates, time-to-revert, and discussion patterns for newcomers vs. experienced editors. Deliver a reproducible dataset, fairness-style metrics, and a dashboard highlighting where newcomer edits are disproportionately reverted.

### Action Items

**If this is the beginning (research & design)**

* Scope pages: 100–300 English Wikipedia articles across elections, immigration, policing, public health, human rights, climate; save as `seed_pages.csv`.
* Define “newcomer”: editor whose **account age < 30 days** at the time of edit **or** not autoconfirmed; treat anonymous/IP editors as a separate cohort.
* Metrics: per-page/month **revert rate**, **time-to-revert**, **edit survival @ {1,7,30} days**, **share of talk-page replies** to newcomers, and **delta** vs experienced editors.
* Methods: revision stream (`prop=revisions`), user metadata (`list=users` with `registration|groups|editcount`), revert detection (SHA1 revert-to + tags/comments), talk page activity (`Talk:` revisions).
* Tooling (pick pairs): `requests` or `httpx`; `pandas` or `polars`; storage `duckdb` or `sqlite`; viz `altair` or `plotly`.

**If researched and ready (implementation steps)**

1. **Seed & resolve**

* Resolve `pageid` for titles; capture redirects. Persist `(pageid,title,topic)`.
2. **Pull revision streams**

* `prop=revisions&rvprop=ids|timestamp|user|userid|sha1|size|comment|tags&rvlimit=max` for each page across 2019→present (or chosen window); handle continuation.
3. **User metadata & cohorts**

* For unique users, call `list=users&ususers=<batch>&usprop=registration|groups|editcount`.
* Label **newcomer** if `edit_ts - registration_ts < 30 days` or user not in autoconfirmed groups (fallback). Keep **anonymous** as its own cohort.
4. **Revert detection**

* SHA1 equality to detect full reverts (revert-to).
* Comment/tag heuristic `(revert|rv|undid|rollback)` (case-insensitive) for partial reverts.
* Build events `(pageid, ts, reverter → reverted, type)`.
5. **Talk-page context**

* For each article’s `Talk:<title>`, aggregate monthly edits & unique editors; attribute replies to newcomer edits by proximity (±48h window heuristic).
6. **Features & KPIs**

* Per page/month and cohort: edits, reverts, **revert rate**, **median time-to-revert**, survival @ {1,7,30} days, talk-page edits.
* Fairness deltas: newcomer vs experienced differences; flag pages with large, persistent gaps.
7. **Deliver**

* Artifacts: `revisions_raw.parquet`, `users.parquet`, `reverts.parquet`, `talk_monthly.parquet`, `page_monthly.parquet`, `metrics.csv`.
* Dashboard: timelines, cohort comparisons, page rankings by newcomer-revert gap.
* Methods README with assumptions and caveats.
8. **Quality & Ops**

* Caching; retries with exponential backoff; honor `maxlag`; persist `continue` tokens.
* Tests: pagination continuity, cohort labeling, SHA1 revert detection, comment regex.
* Error handling: `try/except` with clear `print()` for I/O; **raise with trace** on dtype mismatches; warn on recoverable issues.
* Optional: scheduled monthly refresh (GitHub Actions).

### Resources/Instructions

**API docs to pin in repo**

```
Action API overview: https://www.mediawiki.org/wiki/API:Action_API
Revisions (timestamps, users, sha1, tags): https://www.mediawiki.org/wiki/API:Revisions
Users (registration, groups, editcount): https://www.mediawiki.org/wiki/API:Users
Query & continuation etiquette (maxlag): https://www.mediawiki.org/wiki/API:Query
```

**Suggested libraries (choose pairs)**

* HTTP: `requests` | `httpx`
* DataFrames: `pandas` | `polars`
* Storage: `duckdb` | `sqlite`
* Viz: `altair` | `plotly`

**Sample queries (copy to notes)**

```
# Resolve pageids
action=query&titles=<TITLE>

# Full revision stream (use continuation)
action=query&prop=revisions&titles=<TITLE>&rvprop=ids|timestamp|user|userid|sha1|size|comment|tags&rvlimit=max

# Batch user metadata (to label newcomers)
action=query&list=users&ususers=<USER1>|<USER2>|...&usprop=registration|groups|editcount

# Talk page revisions (oversight/discussion proxy)
action=query&prop=revisions&titles=Talk:<TITLE>&rvprop=timestamp|user|comment|tags&rvlimit=max
```

**Ethics & caveats**

* Aggregate reporting; no editor-level callouts.

* Registration timestamps can be missing for older accounts; fall back to groups + anon heuristic and document missingness.

* SHA1 detects exact reverts; partial reverts rely on comment/tag heuristics (expected FN/FP).

* If this issue requires access to 311 data, please answer the following questions:

* Not applicable.
* N/A
* N/A
* N/A

---

## Project Outline (detailed plan for this idea) in details:

<details>

**Research question**
Are newcomer edits on sensitive pages reverted more often or faster than experienced editors’ edits, and where are the largest, persistent gaps?

**Data sources & modules**

* `prop=revisions` for edit streams.
* `list=users` for registration, groups, editcount.
* `Talk:` page revisions for discussion proxies.

**Method**

1. Build the page cohort and pull revision streams (article + talk).
2. Label editor cohorts using registration age (<30d at edit time), groups, and anon flag.
3. Detect reverts (SHA1 + tag/comment).
4. Compute monthly per-cohort metrics per page (revert rate, time-to-revert, survival).
5. Rank pages by newcomer-experienced gaps; visualize trends and flag persistence.

**Key metrics**

* Newcomer vs experienced **revert rate** and **median time-to-revert**.
* Edit survival @ {1,7,30} days by cohort.
* Talk-page engagement around newcomer edits (±48h).
* Persistence: % months where newcomer gap is in top decile.

**Deliverables**

* Clean tables (`revisions_raw.parquet`, `users.parquet`, `reverts.parquet`, `page_monthly.parquet`).
* Notebook + `reports/newcomer_burden.md`.
* Streamlit/Altair dashboard (cohort comparison, page rankings, timelines).

**Limitations**

* Missing registration data and imperfect revert heuristics; document coverage and validate on a sampled set.
* High-volume pages → large payloads; consider windowing or top-N by recency.

</details>