MediaWiki API Project: Gendered/Respectful Language Drift (think pronouns) #234

Overview

Track gendered/respectful language drift on Wikipedia by measuring how usage of a curated term list (deprecated → preferred) changes over time on policy-relevant pages. Deliver a reproducible dataset, statistical deltas, and a small dashboard highlighting rise/fall, first-adoption dates, and context snippets.

Action Items

If this is the beginning (research & design)

• Define scope: English Wikipedia, Namespace 0; 100–300 pages across immigration, policing, elections, public health, and human rights (seed_pages.csv with title,pageid,qid).
• Build terms.csv: columns deprecated,preferred,aliases,regex,notes (e.g., whitelist→allowlist, master/slave→primary/replica, illegal immigrant→undocumented immigrant). Include canonicalization rules (hyphen/space variants).
• Choose windows: monthly sampling from 2016 → present (adjustable).
• Metrics: term frequency per page/month, log-odds (with informative Dirichlet prior) past vs recent, first-seen date for preferred terms, disappearance of deprecated terms, context windows (±25 tokens).
• Tooling (pick one per pair): requests or httpx; pandas or polars; mwparserfromhell or wikitextparser; storage duckdb or sqlite; viz altair or plotly.

If researched and ready (implementation steps)

1. Seed & resolve

   • Resolve pageid via action=query&titles=<title>; store redirects.

2. Revision sampling

   • For each page/month, fetch the oldest revision in the month (creation if earlier) and the latest revision (prop=revisions&rvprop=ids|timestamp&rvlimit=max to list; then pull rvslots=main&rvprop=content for targeted revids).

3. Text extraction & normalization

   • Convert wikitext → plain text; drop templates, infoboxes, references, code/pre blocks, and quotes where feasible; lowercase + Unicode normalize; apply alias/regex normalization from terms.csv.

4. Counting & context

   • Count deprecated/preferred terms; capture context windows (±25 tokens) for QA; flag code/citation hits for exclusion.

5. Analysis

   • Compute per-term Δfrequency, Δshare (preferred / (preferred+deprecated)), and log-odds change between early (e.g., 2016–2018) and recent (last 12 months). Identify newly adopted (now>0, then=0) and disappeared (then>0, now=0) terms.

6. Deliver

   • Artifacts: page_month_samples.parquet, term_counts.parquet, term_context.parquet, metrics.csv.
   • Dashboard: rising/falling terms, adoption timelines, page-level diff view with context snippets.
   • Methods README (API params, sampling rationale, exclusions, limitations).

7. Quality & Ops

   • Caching + retries with exponential backoff; honor maxlag; persist continue tokens.
   • Tests: regex correctness, sampler month coverage, exclusion filters, deterministic counts on a fixed snapshot.
   • Optional monthly refresh via GitHub Actions.

Resources/Instructions

API docs to pin in repo

MediaWiki Action API: https://www.mediawiki.org/wiki/API:Action_API
Query & continuation: https://www.mediawiki.org/wiki/API:Query
Revisions (timestamps, content): https://www.mediawiki.org/wiki/API:Revisions
Pages → pageids: https://www.mediawiki.org/wiki/API:Query#By_page_title

Suggested libraries (choose pairs)

• HTTP: requests | httpx
• DataFrames: pandas | polars
• Parsing: mwparserfromhell | wikitextparser
• Storage: duckdb | sqlite
• Viz: altair | plotly

Sample queries to copy into notes

# Resolve pageids from titles
action=query&titles=<TITLE>

# List revisions to find oldest/newest in a month (use continuation)
action=query&prop=revisions&rvprop=ids|timestamp&rvlimit=max&titles=<TITLE>

# Fetch content for a specific revision id
action=query&prop=revisions&revids=<REVID>&rvslots=main&rvprop=content

Data handling & ethics

• Report aggregates; avoid editor-level profiling.

• Exclude quotes, code, and citation/reference sections to reduce false positives.

• Terms can be context-sensitive; keep a manual review sheet for ambiguous hits.

• Error handling: wrap I/O in try/except with clear print() messages for file-not-found; terminate with a trace on dtype mismatches; log warnings for recoverable issues.

• If this issue requires access to 311 data, please answer the following questions:

  • Not applicable.
  • N/A
  • N/A
  • N/A

Project Outline (detailed plan for this idea) in details: