MediaWiki API Project: Language Equity (Foreign Languages)

Overview

Measure language equity on sensitive topics by comparing how quickly key Wikipedia pages are created and updated across languages. Output a reproducible dataset and dashboard that quantify coverage (existence), “time-to-translation,” and update lag versus a reference language (e.g., English).

Action Items

If this is the beginning (research & design)

Define scope: start with ~50 English Wikipedia pages in sensitive domains (public health, migration, elections, climate disasters, human rights).
Choose language set: top 30 Wikipedias by article count + 10 low-resource languages for contrast (via sitematrix), or all languages returned by each page’s langlinks.
Finalize metrics & windows: coverage status, time-to-first-presence (page exists Y/N and when), and update lag (delta between latest edit timestamps across languages).
Methods plan: use MediaWiki Action API (prop=langlinks, prop=revisions, list=sitematrix) and optionally Pageviews REST API for context; decide reference language(s) for lags.
Tooling pairs: requests or httpx; pandas or polars; storage in duckdb or sqlite; viz in Altair or Plotly.

If researched and ready (implementation steps)

Seed topics
- Create a topics.csv of English page titles and (optional) Wikidata QIDs for disambiguation.
Enumerate languages & interlanguage links
- For each English title: prop=langlinks to get language codes and titles; merge with sitematrix to validate wiki domains.
Fetch revision metadata
- For English and each linked language title: prop=revisions&rvprop=timestamp|size|userid|comment&rvlimit=1&rvdir=older/newer as needed to get first and latest revision timestamps.
Compute metrics
- Coverage: page present (1/0).
- Time-to-presence: first non-English creation time minus English creation time.
- Update lag: latest English edit time minus latest non-English edit time (days).
- Optional robustness: compare against median of top-N languages instead of only English.
Deliver
- Artifacts: pages_raw.parquet, lang_presence.parquet, lags.parquet, metrics.csv.
- Dashboard: heatmap (languages × topics) of update lags; coverage bar charts; language ranking tables.
- Methods README with exact queries, rate-limit/backoff notes, and caveats.
Quality & Ops
- Caching of responses; exponential backoff on maxlag.
- Unit tests for merge logic and timestamp math; schema checks.
- (Optional) Monthly GitHub Action to refresh a subset.

Instructions

API docs to pin in the repo

MediaWiki Action API overview: API:Action_API
Interlanguage links: API:Langlinks
Revisions metadata: API:Revisions
Site matrix (language list): API:Sitematrix
Pageviews (REST; optional context): Wikimedia REST API: Pageviews

Suggested libraries (pick pairs)

HTTP: requests | httpx
Frames: pandas | polars
Storage: duckdb | sqlite
Viz: Altair | Plotly
Parsing (optional): mwparserfromhell | wikitextparser

Sample queries to copy into notes

# Interlanguage links for an English page
action=query&prop=langlinks&titles=Migration_crisis&lllimit=max

# Latest revision timestamp for a given title on any wiki
action=query&prop=revisions&rvprop=timestamp|size&rvlimit=1&titles=<TITLE>

# First revision timestamp (creation): request oldest
action=query&prop=revisions&rvprop=timestamp&rvlimit=1&rvdir=newer&titles=<TITLE>

# Languages list (code ↔ wiki mapping)
action=sitematrix&formatversion=2

Ethics & reporting

Avoid punitive “league tables.” Emphasize resource constraints and volunteer capacity.
Aggregate results; no profiling of individual editors.
Document missingness (pages that never existed, renamed pages, disambiguation).
Be explicit that latest edit time ≠ content parity (it’s a proxy).
If this issue requires access to 311 data, please answer the following questions:
- Not applicable.
- N/A
- N/A
- N/A

Project Outline (detailed plan for this idea) in details below:

Details

Research question
Do sensitive-topic pages appear and stay up-to-date across languages at similar speeds, or do we observe systematic coverage gaps and update lags?

Data sources & modules

prop=langlinks to discover cross-language equivalents for each English seed page.
list=sitematrix to enumerate languages and validate wiki domains.
prop=revisions to get first and latest timestamps per page/language.
(Optional) Pageviews REST API to contextualize demand vs. freshness.

Method

Build a seed list of English pages in public health, migration, elections, climate disasters, and human rights (store in topics.csv).
For each page, pull langlinks to get target language titles; validate with sitematrix.
For English + each language title, fetch first and latest revision timestamps.
Compute metrics per (topic, language): coverage, time-to-presence, update lag; summarize by language family/region.
Visualize a lag heatmap (languages × topics), coverage distributions, and top-lagging topics.

Key metrics

Coverage rate (% of topics that exist per language).
Median time-to-presence (days).
Median update lag (days) and % of topics with lag > thresholds (e.g., >30, >90 days).
(Optional) Correlate lag with pageviews to see “high-demand but stale” cases.

Deliverables

Clean tables (lang_presence.parquet, first_latest_revisions.parquet, lags.parquet).
Reproducible notebook + reports/language_equity.md.
Streamlit/Altair dashboard with filters (topic set, language subset, thresholds).

Caveats & limitations

Timestamp proxies don’t guarantee semantic parity; translations may be partial.
Some languages may title pages differently or merge topics; handle redirects carefully.
API throttling and maxlag require polite batching and retries.

Implementation notes

Normalize timestamps to UTC; compute diffs in days (float).
Use stable keys: (wiki_db, pageid) when available; fall back to (lang_code, normalized_title).
Cache raw JSON responses and write a manifest of query params for reproducibility.

Resources

Instructions

Progress: "What is the current status of your project? What have you completed and what is left to do?"
Blockers: "Difficulties or errors encountered."
Availability: "How much time will you have this week to work on this issue?"
ETA: "When do you expect this issue to be completed?"
Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

Progress:
- Completed: Data collection (2,240 records: 56 topics × 40 languages), analysis pipeline, 6 interactive visualizations, technical documentation
- Key Results: Quantified 23% coverage gap and 3.2x update lag for low-resource languages; identified critical gaps in health topics (Ebola, Monkeypox 97.5% missing)
- Left to do: Final review and submission
Blockers:
None. All technical issues resolved.
Availability:
3-5 hours available for revisions and feedback as needed.
ETA:
Complete. Ready for review (January 26, 2026).
Pictures:
- Summary dashboard showing language inequality patterns
  <img width="1676" height="1000" alt="newplot" src="

Warning

This asset could not be copied from your saved reply. Please try again later.

" />

Progress: I'm reading on Jupyter Notebook best practices so that I can make the notebook as reproducible as possible.
Blockers: None at the moment, but I've had intermittent issues with the Jupyter kernel not being recognized by VS Code
Availability: 10 hours
ETA: 07/13/2026
Pictures: N/A

Progress: I've migrated the logic from my earlier iterations into a Jupyter notebook and created an accompaning utils file. My goal was to improve the modularity and maintainability of the codebase so that it is easier to debug and reproduce. The notebook now contains explanations of the code and data flow, and the utils file contains core operations for data processing and storage, API requests, and response caching. I've also incorporated the visualization logic from my earlier iterations, which now pulll from the dataframes built off the parquet. I wrote an expansive topics list so that I can decide which ones I want to keep after the run but averaging for 50 topics, the runtime is around 1 hour and 32 minutes.
Blockers: Time constraint - I didn't have as much time as I thought I would last week and the progress I made took longer than I expected.
Availability: ~5 hours
ETA: 08/03/2026

Next steps: I will cut down the topic list to 50 topics and organize them by domain, before I continue working on the visualizations. I plan to have everything ready for our next meeting and will report progress weekly.

Progress: I've completed the data collection pipeline, computed the metrics, and created the visualizations for the project. The final dataset (50 core topics comprehensively mapped across 300+ Wikipedia language editions) has been successfully generated and cached. I've also optimized the API fetching logic, which previously took over 1.5 hours, and now executes concurrently in roughly 2 minutes without hitting any rate limits.
Blockers: None
Availability: ~5 hours
ETA: 08/03/2026

Next steps:
The core project requirements are complete! My remaining tasks are to apply the finishing touches to the visualizations, draft the final language_equity.md report, and clean up the Jupyter notebook for final delivery.

Progress: I am still applying the finishing touches and drafting the report
Blockers: I had a couple of emergencies the last couple of weeks, so I didn't get a chance to work on the issue
Availability: Unsure this week
ETA: 08/20/2026
Pictures:

Progress: I left off in the same place as last week
Blockers: I didn't have any time to work on it last week
Availability: Unsure but hopefully around 10 hours
ETA: 08/24/2026
Pictures:

@egcuriel

Please provide update @egcuriel

Instructions

Progress: "What is the current status of your project? What have you completed and what is left to do?"
Blockers: "Difficulties or errors encountered."
Availability: "How much time will you have this week to work on this issue?"
ETA: "When do you expect this issue to be completed?"
Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

You can use this template

1. Progress: 
2. Blockers: 
3. Availability:
4. ETA:
5. Pictures (if necessary):

Progress: I've left off on the same as last week since I didn't have much time to work on things last week
Blockers: Time
Availability: Unsure
ETA: Unsure
Pictures:

@egcuriel

Please provide update @egcuriel

Instructions

Progress: "What is the current status of your project? What have you completed and what is left to do?"
Blockers: "Difficulties or errors encountered."
Availability: "How much time will you have this week to work on this issue?"
ETA: "When do you expect this issue to be completed?"
Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

You can use this template

1. Progress: 
2. Blockers: 
3. Availability:
4. ETA:
5. Pictures (if necessary):

Progress: I am working on the report at the moment
Blockers: Time has been very limited on my end
Availability: Sat-Sun, 9pm-11pm
ETA: 09/07/2026

@egcuriel

Please provide update @egcuriel

Instructions

Progress: "What is the current status of your project? What have you completed and what is left to do?"
Blockers: "Difficulties or errors encountered."
Availability: "How much time will you have this week to work on this issue?"
ETA: "When do you expect this issue to be completed?"
Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

You can use this template

1. Progress: 
2. Blockers: 
3. Availability:
4. ETA:
5. Pictures (if necessary):

Progress: I've migrated everything over to be contained in a data and scripts folder. The scripts folder now contains scripts for their corresponding scope (models, network, api, pipeline, analytics, visualizations, utils). The data folder contains .cache, logs, visualizations folders and the csv and parquet files.
Blockers:
Availability: M-F (6pm -11pm)
ETA: 09/21/2026
Pictures: None

Next Steps: I'm going to finish off the report with the updated visualizations and an analysis of the findings

Progress:

API calls
I realized that making individual API calls for translated title can be mitigated by:

fetch reference title by calling WikiData API -> get langlinks
organize the list by language -> pipe titles to make one call to MediaWiki API to its respective domain
use revids from each article and pipe them to fetch latest revisions from MediaWiki API

An individual call still needs to be made for the first revision, but at least the proceding calls such as the latest revisions don't need to be made individually every time they must be updated.

Storage

Making one call per language does mean that retrieving cache is more involved since the urls are now for a batch of articles
I've worked in looping through the cache directory for every API call, but this might not be the most efficient or practical choice, so im considering putting the titles and hashed url in a separate table instead.

Next steps

I know there are libraries that simplify various steps, but in the sake of minimizing dependencies, I thought I'd give it a shot before I go that route.
I will rework the pipeline with the new fetching and cache retrieving structure

@egcuriel

Please provide update @egcuriel

Instructions

Progress: "What is the current status of your project? What have you completed and what is left to do?"
Blockers: "Difficulties or errors encountered."
Availability: "How much time will you have this week to work on this issue?"
ETA: "When do you expect this issue to be completed?"
Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

You can use this template

1. Progress: 
2. Blockers: 
3. Availability:
4. ETA:
5. Pictures (if necessary):

Progress: I've made a couple of changes since our last meeting regarding the structure and how the pipeline handles API requests, caching, and storage:
API
- I've decided to use the MediaWiki API and query for langlinks. Although querying Wikidata directly for QIDs and revids might have meant fewer calls, capturing how every language assigns QIDs and handles local redirects is valuable data to preserve.
- I structured the extraction into four batched passes (seed metadata, langlinks, foreign metadata, and first revisions) up to 50 titles per request, interleaving foreign language requests to distribute traffic. The client also handles switching from GET to POST when encountering 414 URL length limits and applies backoff when hit with MediaWiki lag errors.
Cache
- The batched responses are compressed and saved as .json.gz files using an MD5 hash of the normalized URL and sorted parameters as the filename.
- The hash folder is partitioned across 256 directories using the first two characters of the key. This is defensive for the amount of titles I have so far, but it avoids filesystem lookup bottlenecks as the dataset grows or when querying deltas long term.
- Cache writes are handled atomically using thread-isolated temp files before renaming, and the hash is logged alongside its URL and request parameters in a daily JSONL manifest file to track the exact request parameters and origin of each cached file.
Storage
- Ingests cache files directly into DuckDB using its native C++ read_json engine, which bypasses Python-level gzip decompression and JSON deserialization overhead entirely.
- DuckDB stages the .json.gz files directly in memory and unnests them into normalized relational tables (pages, revisions, redirects, langlinks), while storing the source hash on each record so any row can be traced back to its raw API payload.
Blockers: None, although researching each change has taken some time.
Availability: ~ 5 hours
ETA: 11/02/2026
Pictures (if necessary):

Progress: I’m refactoring and comparing class-based and function-based implementations across the API, cache, database, and pipeline/notebook layers. The goal is to determine where classes improve state management and reuse, and where standalone functions provide a simpler design without unnecessary overhead.
Blockers: My laptop has been overheating, so I spent most of the week looking into what it could be.
Availability: 5 hours
ETA: 11/02/2026

Navigation Menu

Repository navigation

MediaWiki API Project: Language Equity (Foreign Languages) #229

Overview

Action Items

Instructions

Project Outline (detailed plan for this idea) in details below:

Resources

14 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

MediaWiki API Project: Language Equity (Foreign Languages) #229

Description

Overview

Action Items

Instructions

Project Outline (detailed plan for this idea) in details below:

Resources

Activity

LixinTu commented on Jan 26, 2026

14 remaining items

egcuriel commented on Jul 7, 2026

egcuriel commented on Jul 14, 2026

egcuriel commented on Jul 22, 2026

egcuriel commented on Aug 14, 2026

egcuriel commented on Aug 18, 2026

chinaexpert1 commented on Aug 25, 2026

egcuriel commented on Aug 25, 2026

chinaexpert1 commented on Sep 1, 2026

egcuriel commented on Sep 5, 2026

chinaexpert1 commented on Sep 22, 2026

egcuriel commented on Sep 22, 2026

egcuriel commented on Sep 29, 2026

chinaexpert1 commented on Oct 20, 2026

egcuriel commented on Oct 20, 2026

egcuriel commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions