Skip to content

MediaWiki API Project: Controversy and Protection Moderation #230

Open
Open
MediaWiki API Project: Controversy and Protection Moderation#230
Assignees
noneill256
Labels
CoP: Data Sciencecomplexity: mediumfeature: missingthis tags is mutually exclusive with project: missing. Please use the correct labelgood first issueGood for newcomersproject: EDArole: data analysisrole: data sciencerole: missingsize: 3ptCan be done in 13-18 hourssize: missing

Description

@chinaexpert1
chinaexpert1
opened on Sep 13, 2025

Overview

Quantify controversy & protection dynamics on policy-sensitive Wikipedia pages by tracking protection events (protect/unprotect, semi/full) and correlating them with edit/revert bursts over time. Deliver a reproducible dataset and dashboard that surface where, when, and how protection is used as a moderation tool.

Action Items

If this is the beginning (research & design)

  • Define scope: 50–200 English Wikipedia pages in domains like elections, policing, immigration, reproductive rights, public health, climate, and disinformation (store in seed_pages.csv with title,pageid,qid if available).
  • Metrics & windows: protection type/level, duration, frequency (events/page/year), time-to-protection after burst, revert ratio before/after protection, unique editors, and edit volume ±{7,14,30} days around events.
  • Methods: MediaWiki Action API modules — list=logevents (protection history), prop=info&inprop=protection (current state), prop=revisions (edit stream with timestamps, users, comments, sha1). Optional: Pageviews REST for demand context; ORES (if enabled) for damaging/goodfaith signals.
  • Tooling (choose pairs and keep consistent): requests or httpx; pandas or polars; duckdb or sqlite; altair or plotly.
  • Ethics: aggregate results; do not profile individual editors; document missing/hidden logs; avoid normative judgments.

If researched and ready (implementation steps)

  1. Seed & resolve pages

    • Ingest seed_pages.csv; resolve pageid via action=query&titles=<title> if missing; record redirects.

  2. Protection history

    • Pull list=logevents with letype=protect|modify|unprotect for each page; capture timestamp, level (e.g., semi/full), expiry, and reason; paginate via continue.
    • Snapshot current protection via prop=info&inprop=protection.

  3. Revision stream (context windows)

    • For each protection event, fetch revisions in windows (e.g., −30 to +30 days) via prop=revisions&rvprop=ids|timestamp|user|comment|sha1|size.
    • Compute per-day edits, unique editors, revert flags (sha1 repeat, or comment contains “revert/undid/rv”), and size deltas.

  4. Feature engineering

    • Event-level table: pageid, event_ts, level, duration, edits_before/after, revert_ratio_before/after, Δunique_editors, peak_day_edits, time_to_peak, current_protection_state.

  5. Analysis

    • KPIs: events/page/year; median protection duration; % semi vs full; typical time-to-protection after spike; change in revert ratio post-protection; pages with recurring protections (re-protect within 90 days).
    • Optional: structural-break tests on edit counts; correlate with pageviews.

  6. Deliver

    • Artifacts: protection_events.parquet, revision_windows.parquet, metrics.csv.
    • Dashboard: timelines with event overlays; small-multiples per topic; tables of top pages by protection frequency/duration.
    • Methods README: exact queries, rate-limit/backoff strategy, assumptions, and limitations.

  7. Quality & Ops

    • Caching, retries with exponential backoff; log maxlag responses; store raw JSON.
    • Unit tests: pagination correctness, windowing logic, revert-detection heuristics.
    • Optional: monthly GitHub Action to refresh new events.

Resources/Instructions

Docs & endpoints to pin in repo

MediaWiki Action API overview: https://www.mediawiki.org/wiki/API:Action_API
Logevents (protect/unprotect): https://www.mediawiki.org/wiki/API:Logevents
Page info & protection: https://www.mediawiki.org/wiki/API:Info
Revisions (timestamps, sha1, comments): https://www.mediawiki.org/wiki/API:Revisions
Page titles → pageids: https://www.mediawiki.org/wiki/API:Query
(Optional) Pageviews REST: https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews

Suggested libraries (pick pairs)

  • HTTP: requests | httpx
  • Frames: pandas | polars
  • Storage: duckdb | sqlite
  • Viz: altair | plotly
  • Parsing (optional): mwparserfromhell | wikitextparser

Sample queries to copy into notes

# Protection history for a page
action=query&list=logevents&letype=protect|modify|unprotect&lelimit=max&leprop=title|timestamp|details|comment&leend=<ISO_END>&lestart=<ISO_START>&letitle=<PAGE_TITLE>

# Current protection snapshot
action=query&prop=info&inprop=protection&titles=<PAGE_TITLE>

# Latest N revisions in a window
action=query&prop=revisions&rvprop=ids|timestamp|user|comment|sha1|size&rvstart=<ISO_END>&rvend=<ISO_START>&rvlimit=max&titles=<PAGE_TITLE>

Data handling notes

  • Use polite batching and honor maxlag; persist continue tokens; store raw responses.

  • Prefer pageid as key; handle renames/redirects. Normalize timestamps to UTC.

  • Revert heuristic: sha1 repeat or comment regex (revert|rv|undid|undo) (case-insensitive). Tune and document.

  • If this issue requires access to 311 data, please answer the following questions:

    • Not applicable.
    • N/A
    • N/A
    • N/A

Project Outline (detailed plan for this idea) in details below:

Details

Research question
When do sensitive-topic pages get protected, for how long, and what editing patterns (edits, reverts, editor counts) precede and follow those protections?

Data sources & modules

  • list=logevents with letype=protect|modify|unprotect for event history.
  • prop=info&inprop=protection for current state.
  • prop=revisions for edit streams (timestamps, sha1, comments, size).
  • (Optional) Pageviews REST for demand context.

Method

  1. Build a topic seed list and resolve pageid.
  2. Extract all protection events (timestamps, level, expiry) and derive durations (until unprotect/expiry).
  3. For each event, pull revision windows (e.g., ±30 days), compute daily edits, unique editors, revert ratio, and size change.
  4. Aggregate to page-level and topic-level metrics; identify recurrent protection patterns and median “time-to-protection” after spikes.
  5. Visualize timelines with overlays and produce ranked tables.

Key metrics

  • Events/page/year; median/mean protection duration; % semi/full; re-protect within 90 days.
  • Δrevert ratio pre→post; Δunique editors; time from spike to protection.
  • Pages in top decile by protection frequency or duration.

Deliverables

  • protection_events.parquet, revision_windows.parquet, page_metrics.parquet.
  • Reproducible notebook + reports/controversy_protection.md.
  • Streamlit/Altair dashboard with filters (topic, date range, protection level).

Caveats & limitations

  • Some logs/details may be hidden or suppressed; treat as missing.
  • Protection status ≠ resolution of controversy; it’s a moderation proxy.
  • Revert heuristics are imperfect; validate on a sampled set.

Implementation notes

  • Robust pagination (continue), retry/backoff with prints on failures.
  • Keep a query manifest (params + timestamps) for provenance.
  • Test window slicing and duration calculations; ensure correct handling of overlapping protection intervals.

Activity

  1. chinaexpert1
    added this to CoP: Data Science: Project Boardon Sep 13, 2025
  2. chinaexpert1
    added a parent issue on Sep 13, 2025
  3. github-project-automation
    moved this to New Issue Approval in CoP: Data Science: Project Boardon Sep 13, 2025
  4. ExperimentsInHonesty
    added
    role: missing
    feature: missingthis tags is mutually exclusive with project: missing. Please use the correct label
    size: missing
    on Oct 9, 2025
  5. chinaexpert1
    added
    good first issueGood for newcomers
    role: data science
    role: data analysis
    size: 3ptCan be done in 13-18 hours
    project: EDA
    CoP: Data Science
    complexity: medium
    on Feb 10, 2026
  6. noneill256
    self-assigned this
    on Apr 28, 2026

  7. noneill256 commented on May 10, 2026

    @noneill256
    noneill256
    on May 10, 2026
    Member
    Instructions
    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."
    1. Progress: Became familiar with using the Wikimedia API in Python. Started to create my list of controversial Wikipedia pages to analyze.
    2. Blockers: None so far, though I'm wondering if my list is similar enough to what was imagined when this project was first come up with. Is it too US-centric / too focused on events of recent years, or does that matter?
    3. Availability: 5 hours
    4. ETA: The list should be finished by the next meeting, and I will hopefully have started implementing the pages into Python as well.

  8. 2 remaining items

  9. noneill256 commented on Jun 1, 2026

    @noneill256
    noneill256
    on Jun 1, 2026
    Member
    Instructions
    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."
    1. Progress: Finished the QID-matching segment of the script. Pulling all revision data as well as multiple categories with this API does not work quite like how I expected, so I was unable to make much progress with that.
    2. Blockers: Unsure of the article categories I've chosen, would like to go over in meeting. Also some articles don't show complete revision data.
    3. Availability: 4 hours
    4. ETA: Plan to pull at least page IDs and hopefully QIDs for all the chosen categories. Should be able to save to a CSV as well. Hoping to figure out how to pull revision data as well.

  10. noneill256 commented on Jun 16, 2026

    @noneill256
    noneill256
    on Jun 16, 2026
    Member

    Duplicate of #

    Instructions

    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."
    1. Progress: Script can now pull all page IDs and QIDs for pages.
    2. Blockers: MediaWiki API doesn't allow the pulling of every single article under the "History of [Continent]" like I expected. Will be proceeding with a handmade list of articles to analyze instead.
    3. Availability: 5 hours
    4. ETA: Will write code to pull revision data for each page and possibly begin high level analysis
    5. Pictures (if necessary):

  11. noneill256 commented on Jun 23, 2026

    @noneill256
    noneill256
    on Jun 23, 2026
    Member
    Instructions
    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."
    1. Progress: Expanded list of pages to analyze in google sheets. Wrote the script to pull protection data for each page.
    2. Blockers: Unsure of how exactly to format the revision data
    3. Availability: 5 hours
    4. ETA: Unsure of next steps, but will have data in a different format by next week and will also include protection data
    5. Pictures (if necessary):

  12. noneill256 commented on Jul 14, 2026

    @noneill256
    noneill256
    on Jul 14, 2026
    Member
    Instructions
    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."
    1. Progress: Didn't get much done, took a long time to figure out how to pull protection data.
    2. Blockers: Not entirely sure of how to interpret protection data. Concerned that maybe there isn't enough of it.
    3. Availability: 5 hours
    4. ETA: Current goal for next week is to learn more about protection data and also reformat data to have each event be a single row, rather than conglomerated by page.
    5. Pictures (if necessary):

  13. noneill256 commented on Aug 17, 2026

    @noneill256
    noneill256
    on Aug 17, 2026
    Member
    Instructions
    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."
    1. Progress: I've extracted all the necessary data for my list of pages. The revisions data is nested, with 500 revisions in each row, and needs to be extracted into multiple rows.
    2. Blockers: There are 1.2 million revisions. I tried to extract them into individual rows, but even after running this code for multiple hours, it didn't get through every revision.
    3. Availability: 4 hours
    4. ETA: Hoping to be able to unnest these revisions by next week

  14. chinaexpert1 commented on Aug 25, 2026

    @chinaexpert1
    chinaexpert1
    on Aug 25, 2026 · Hidden as resolved
    Author

  15. noneill256 commented on Aug 25, 2026

    @noneill256
    noneill256
    on Aug 25, 2026
    Member
    1. Progress: Made next to no progress while messing around with ways to reduce the number of rows. I may need to use pySpark, which I'll ask questions about in our meeting.
    2. Blockers: Still too many rows to extract
    3. Availability: 4 hours
    4. ETA: No idea, may need to start learning Spark

  16. noneill256 commented on Sep 1, 2026

    @noneill256
    noneill256
    on Sep 1, 2026
    Member
    Instructions
    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."
    1. Progress: Didn't have time to do much this week, but decided to trim down the amount of data I pull by just filtering to edits from the year 2012 and after, which has cut down my data by half.
    2. Blockers: None
    3. Availability: 4 hours
    4. ETA: Hoping to have the nested data extracted and ready for analysis by next meeting

  17. chinaexpert1 commented on Sep 15, 2026

    @chinaexpert1
    chinaexpert1
    on Sep 15, 2026 · Hidden as resolved
    Author

  18. noneill256 commented on Sep 15, 2026

    @noneill256
    noneill256
    on Sep 15, 2026
    Member
    Instructions
    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."
    1. Progress: Split the raw data into multiple sections so that unnesting it is faster. Data is now ready for analysis.
    2. Blockers: N/a
    3. Availability: 5 hours
    4. ETA: Not sure, could use suggestions for first steps in tonight's meeting
    5. Pictures (if necessary):

  19. chinaexpert1 commented on Sep 22, 2026

    @chinaexpert1
    chinaexpert1
    on Sep 22, 2026
    MemberAuthor

    Please provide update @noneill256

    Instructions
    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

    You can use this template

    1. Progress: 
2. Blockers: 
3. Availability:
4. ETA:
5. Pictures (if necessary):

  20. noneill256 commented on Sep 22, 2026

    @noneill256
    noneill256
    on Sep 22, 2026
    Member
    Instructions
    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."
    1. Progress: Wasn't able to work on this this week.
    2. Blockers: n/a
    3. Availability: 4 hours
    4. ETA: Will start trying to work size deltas into data by next week, and maybe create a day-by-day dataset rather than one with one row per edit.
    5. Pictures (if necessary):

  21. noneill256 commented on Oct 6, 2026

    @noneill256
    noneill256
    on Oct 6, 2026
    Member
    Instructions
    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."
    1. Progress: Added "size" and "size delta" columns to revision dataset. Changed some columns data types.
    2. Blockers: None
    3. Availability: 4 hours
    4. ETA: Would like to move on to analysis, currently thinking of calculating which pages have the most edits per month
    5. Pictures (if necessary):

  22. noneill256 commented on Oct 13, 2026

    @noneill256
    noneill256
    on Oct 13, 2026
    Member
    Instructions
    1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
    2. Blockers: "Difficulties or errors encountered."
    3. Availability: "How much time will you have this week to work on this issue?"
    4. ETA: "When do you expect this issue to be completed?"
    5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."
    1. Progress: Encountered a bug in my unnesting script that's causing certain rows to have empty values in the place of their article title.
    2. Blockers: "
    3. Availability: 4 hours
    4. ETA: Hoping to have this resolved next week so I can start analysis
    5. Pictures (if necessary):

  23. chinaexpert1 commented on Oct 27, 2026

    @chinaexpert1
    chinaexpert1
    on Oct 27, 2026
    MemberAuthor

    Noah is taking amonth off

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

CoP: Data Sciencecomplexity: mediumfeature: missingthis tags is mutually exclusive with project: missing. Please use the correct labelgood first issueGood for newcomersproject: EDArole: data analysisrole: data sciencerole: missingsize: 3ptCan be done in 13-18 hourssize: missing

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions