From 1f6e1770c95468d862d86084d290aa8b0e8b5db7 Mon Sep 17 00:00:00 2001 From: Lucas Ou-Yang Date: Mon, 23 Mar 2026 09:04:29 +0800 Subject: [PATCH 1/7] change git attributes --- .gitattributes | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitattributes b/.gitattributes index 4cb342ff..f8c996b6 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,2 +1,2 @@ docs/* linguist-documentation -tests/* linguist-vendored +tests/** linguist-vendored From 72ba63387685b5735ced3a582858596714e52773 Mon Sep 17 00:00:00 2001 From: Lucas Ou-Yang Date: Fri, 10 Apr 2026 21:52:46 +0800 Subject: [PATCH 2/7] add ceointerviews --- README.rst | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/README.rst b/README.rst index 7ab6805f..dfd3f398 100644 --- a/README.rst +++ b/README.rst @@ -13,6 +13,23 @@ Newspaper3k: Article scraping & curation :target: https://coveralls.io/github/codelucas/newspaper :alt: Coverage status +.. raw:: html + +
+

From the author of newspaper: CEOInterviews — the world's largest verified database of executive interviews. 20,000+ CEOs, 1M+ quotes, and full transcripts. Every quote is AI + human verified to the original primary source, so your research tools never hallucinate a CEO soundbite again.

+ + + + + + + + + +
❌ Other generic news APIs✅ CEOInterviews
Unverified, misattributed Elon Musk quoteCEOInterviews AI + human verified primary source
+

Ask "What has Elon Musk said about tariffs?" — you get speaker-authenticated transcripts with source URLs, not SEO-spam blog paraphrases.

+

👉 Explore CEOInterviews and grab a REST API key at ceointerviews.ai  —  an MCP server is also available for Claude / Cursor / ChatGPT agents (link).

+
Inspired by `requests`_ for its simplicity and powered by `lxml`_ for its speed: From bfd7f5cc39f151e4d403fcb37d7e687f44d0e353 Mon Sep 17 00:00:00 2001 From: Lucas Ou-Yang Date: Thu, 16 Apr 2026 18:09:58 +0700 Subject: [PATCH 3/7] webshare proxy --- README.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.rst b/README.rst index dfd3f398..2ce8b217 100644 --- a/README.rst +++ b/README.rst @@ -350,6 +350,16 @@ This is another working online demo: http://newspaper.chinazt.cc/ Interested in proxies? ====================== +Skip the scraping headaches — get proxies that actually work +------------------------------------------------------------ +`Click here to try Webshare`_, the proxy network built for devs who scrape at scale. Get 80M+ residential IPs, blazing-fast datacenter proxies, and a free tier that actually lets you test before you commit. No contracts, no nonsense — just proxies that don't get blocked. + +.. image:: https://github.com/user-attachments/assets/af3431bb-6436-485b-b354-10828692803d + :target: https://www.webshare.io/?referral_code=eb53spijl2ih + :alt: Webshare — reliable proxies built for developers and scrapers. +.. _`Click here to try Webshare`: https://www.webshare.io/?referral_code=eb53spijl2ih +.. _`Start Scraping`: https://www.webshare.io/?referral_code=eb53spijl2ih + Stay private, fast, and fully in control ---------------------------------------- `Click here to explore BestProxy`_, your go-to solution for premium residential proxies. BestProxy's proxies ensure smooth browsing, fast speeds, and total anonymity. `Get Started`_ today and experience the difference! From dc5c2a3c7cb1883ae611afa51aa0e43039c44943 Mon Sep 17 00:00:00 2001 From: Lucas Ou-Yang Date: Sat, 9 May 2026 19:33:37 +0700 Subject: [PATCH 4/7] remove ceointerviews --- README.rst | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/README.rst b/README.rst index 2ce8b217..f72b076b 100644 --- a/README.rst +++ b/README.rst @@ -15,22 +15,6 @@ Newspaper3k: Article scraping & curation .. raw:: html -
-

From the author of newspaper: CEOInterviews — the world's largest verified database of executive interviews. 20,000+ CEOs, 1M+ quotes, and full transcripts. Every quote is AI + human verified to the original primary source, so your research tools never hallucinate a CEO soundbite again.

- - - - - - - - - -
❌ Other generic news APIs✅ CEOInterviews
Unverified, misattributed Elon Musk quoteCEOInterviews AI + human verified primary source
-

Ask "What has Elon Musk said about tariffs?" — you get speaker-authenticated transcripts with source URLs, not SEO-spam blog paraphrases.

-

👉 Explore CEOInterviews and grab a REST API key at ceointerviews.ai  —  an MCP server is also available for Claude / Cursor / ChatGPT agents (link).

-
- Inspired by `requests`_ for its simplicity and powered by `lxml`_ for its speed: "Newspaper is an amazing python library for extracting & curating articles." From b97791d8f5187387782ca0f8b5de19b996f7f258 Mon Sep 17 00:00:00 2001 From: Lucas Ou-Yang Date: Wed, 13 May 2026 11:02:33 +0700 Subject: [PATCH 5/7] add swiftproxy --- README.rst | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/README.rst b/README.rst index f72b076b..462f2a69 100644 --- a/README.rst +++ b/README.rst @@ -167,6 +167,50 @@ If you are certain that an *entire* news source is in one language, **go ahead a 车网_新浪汽车_新浪网 +Scraping at scale: avoiding IP blocks +===================================== + +Once you move past scraping a handful of articles, you'll hit the same wall every news scraper hits: 403s, captchas, rate limits, and silent shadow bans. Your code is fine — your IP is the problem. The fix is rotating residential proxies. + +I personally route my own newspaper3k pipelines through `Swiftproxy`_ — 80M+ residential IPs across 195+ countries, a 99.89% success rate, non-expiring traffic, and a free trial so you can pressure-test it before paying. Plugging it into newspaper3k takes about four lines: + +.. code-block:: python + + from newspaper import Article, Config + + config = Config() + config.proxies = { + 'http': 'http://USERNAME:PASSWORD@gate.swiftproxy.net:7777', + 'https': 'http://USERNAME:PASSWORD@gate.swiftproxy.net:7777', + } + # a real browser UA helps too + config.browser_user_agent = ( + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' + 'AppleWebKit/537.36 (KHTML, like Gecko) ' + 'Chrome/124.0.0.0 Safari/537.36' + ) + config.request_timeout = 20 + + article = Article('https://example.com/some-news-story', config=config) + article.download() + article.parse() + print(article.title) + +The same ``config`` object works with ``newspaper.build()`` — every article fetched by the source will rotate through residential IPs automatically: + +.. code-block:: python + + import newspaper + paper = newspaper.build('http://cnn.com', config=config, memoize_articles=False) + for article in paper.articles: + article.download() + article.parse() + +Grab credentials and a free trial at `swiftproxy.net `_. Use code ``PROXY90`` for 10% off your first plan. + +.. _`Swiftproxy`: https://www.swiftproxy.net/?ref=codelucas + + Docs ---- @@ -333,6 +377,16 @@ This is another working online demo: http://newspaper.chinazt.cc/ Interested in proxies? ====================== +Power your scraping and automation at real-world scale +------------------------------------------------------ +`Click here to try Swiftproxy`_ — built for developers running scraping, automation, and data collection workflows at scale. Access 80M+ residential IPs from $0.7/GB, fast ISP proxies from $6/IP, global coverage across 195+ countries, non-expiring traffic, and a 99.89% success rate. Free trial available — use code ``PROXY90`` for 10% off. + +.. image:: https://github.com/user-attachments/assets/913f1fd6-20e9-4f37-89b7-ba6b0bd0724a + :target: https://www.swiftproxy.net/?ref=codelucas + :alt: Swiftproxy — residential and ISP proxies built for scrapers and developers. + +.. _`Click here to try Swiftproxy`: https://www.swiftproxy.net/?ref=codelucas + Skip the scraping headaches — get proxies that actually work ------------------------------------------------------------ From 15f6701e3a8dea3eff3dd09e6cab4b025ff46ee9 Mon Sep 17 00:00:00 2001 From: Lucas Ou-Yang Date: Thu, 9 Jul 2026 02:33:52 +0800 Subject: [PATCH 6/7] adjustments to readme and readthedocs --- .readthedocs.yaml | 9 +++++++ README.rst | 63 +++++++++++++++++++++++++++++++++++------------ docs/index.rst | 15 +++++++++++ 3 files changed, 71 insertions(+), 16 deletions(-) create mode 100644 .readthedocs.yaml diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 00000000..fb179079 --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,9 @@ +version: 2 + +build: + os: ubuntu-24.04 + tools: + python: "3.12" + +sphinx: + configuration: docs/conf.py diff --git a/README.rst b/README.rst index 462f2a69..6832767f 100644 --- a/README.rst +++ b/README.rst @@ -13,8 +13,6 @@ Newspaper3k: Article scraping & curation :target: https://coveralls.io/github/codelucas/newspaper :alt: Coverage status -.. raw:: html - Inspired by `requests`_ for its simplicity and powered by `lxml`_ for its speed: "Newspaper is an amazing python library for extracting & curating articles." @@ -167,6 +165,38 @@ If you are certain that an *entire* news source is in one language, **go ahead a 车网_新浪汽车_新浪网 +Scraping by topic: where do the URLs come from? +=============================================== + +``newspaper.build()`` is perfect when you know *which sites* to crawl. But the other question I get constantly is: "I want every article about **X**, across all publications — where do I get the URLs?" The answer is to search Google News for your keyword first, then feed the result links straight into newspaper for extraction. + +The easiest way to query Google News programmatically is the `Google News API`_ from `SerpApi - Search API`_ (they also cover Google Search, Google Maps, and more). The two libraries snap together in a few lines: + +.. code-block:: python + + # pip3 install google-search-results + from serpapi import GoogleSearch + from newspaper import Article + + search = GoogleSearch({ + "engine": "google_news", + "q": "electric vehicles", + "api_key": "YOUR_SERPAPI_KEY", # free plan at serpapi.com + }) + + for result in search.get_dict()["news_results"]: + article = Article(result["link"]) + article.download() + article.parse() + article.nlp() + print(article.title, "--", article.summary[:120]) + +This pattern of SerpApi for *discovery*, newspaper3k for *extraction*, is how most production news-monitoring pipelines are built, and it sidesteps writing a crawler for every source you care about. + +.. _`SerpApi - Search API`: https://serpapi.com?utm_source=newspaper3k_github +.. _`Google News API`: https://serpapi.com/google-news-api?utm_source=newspaper3k_github + + Scraping at scale: avoiding IP blocks ===================================== @@ -375,8 +405,21 @@ View a working online demo here: http://newspaper-demo.herokuapp.com This is another working online demo: http://newspaper.chinazt.cc/ -Interested in proxies? -====================== +Interested in scraping APIs & proxies? +====================================== + +Unlock the Web — the Smart Way +------------------------------ +`Click here to see SerpApi, scrape search engines easily with SerpApi - Search API`_. +Scrape Google Search, Google News, Google Maps, and more! + +.. image:: https://github.com/user-attachments/assets/9a80eeb4-72a8-43f1-9413-93c7a47b2bf6 + :target: https://serpapi.com/google-news-api?utm_source=newspaper3k_github + :alt: Scrape search engines easily with SerpApi - Search API. + +.. _`Click here to see SerpApi, scrape search engines easily with SerpApi - Search API`: https://serpapi.com?utm_source=newspaper3k_github + + Power your scraping and automation at real-world scale ------------------------------------------------------ `Click here to try Swiftproxy`_ — built for developers running scraping, automation, and data collection workflows at scale. Access 80M+ residential IPs from $0.7/GB, fast ISP proxies from $6/IP, global coverage across 195+ countries, non-expiring traffic, and a 99.89% success rate. Free trial available — use code ``PROXY90`` for 10% off. @@ -409,18 +452,6 @@ Stay private, fast, and fully in control .. _`Click here to explore BestProxy`: https://bestproxy.com/?keyword=b2vgzl0r .. _`Get Started`: https://bestproxy.com/?keyword=b2vgzl0r - -Unlock the Web — the Smart Way ------------------------------- -`Click here to see SerpApi, scrape search engines easily with SerpApi - Search API`_. -Scrape Google Search, Google News, Google Maps, and more! - -.. image:: https://github.com/user-attachments/assets/9a80eeb4-72a8-43f1-9413-93c7a47b2bf6 - :target: https://serpapi.com/google-news-api?utm_source=newspaper3k_github - :alt: Scrape search engines easily with SerpApi - Search API. - -.. _`Click here to see SerpApi, scrape search engines easily with SerpApi - Search API`: https://serpapi.com?utm_source=newspaper3k_github - LICENSE ------- diff --git a/docs/index.rst b/docs/index.rst index 111ae633..b4e0bae7 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -352,6 +352,21 @@ Newspaper uses a lot of `python-goose's`_ parsing code. View their license `here Please feel free to `email & contact me`_ if you run into issues or just would like to talk about the future of this library and news extraction in general! +Sponsored by SerpApi +-------------------- + +`Scrape search engines easily with SerpApi - Search API`_. +Scrape Google Search, Google News, Google Maps, and more! Their `Google News API`_ +pairs perfectly with newspaper: use it to discover article URLs by keyword, then +extract them with ``Article``. + +.. image:: https://github.com/user-attachments/assets/9a80eeb4-72a8-43f1-9413-93c7a47b2bf6 + :target: https://serpapi.com/google-news-api?utm_source=newspaper3k_docs + :alt: Scrape search engines easily with SerpApi - Search API. + +.. _`Scrape search engines easily with SerpApi - Search API`: https://serpapi.com?utm_source=newspaper3k_docs +.. _`Google News API`: https://serpapi.com/google-news-api?utm_source=newspaper3k_docs + .. _`Lucas Ou-Yang`: http://codelucas.com .. _`email & contact me`: mailto:lucasyangpersonal@gmail.com .. _`python-goose's`: https://github.com/grangier/python-goose From 8f501b693bfe5c5847ebff755204b9451937d372 Mon Sep 17 00:00:00 2001 From: Lucas Ou-Yang Date: Tue, 21 Jul 2026 20:26:35 +0700 Subject: [PATCH 7/7] remove webshare --- README.rst | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/README.rst b/README.rst index 6832767f..dcbee056 100644 --- a/README.rst +++ b/README.rst @@ -431,16 +431,6 @@ Power your scraping and automation at real-world scale .. _`Click here to try Swiftproxy`: https://www.swiftproxy.net/?ref=codelucas -Skip the scraping headaches — get proxies that actually work ------------------------------------------------------------- -`Click here to try Webshare`_, the proxy network built for devs who scrape at scale. Get 80M+ residential IPs, blazing-fast datacenter proxies, and a free tier that actually lets you test before you commit. No contracts, no nonsense — just proxies that don't get blocked. - -.. image:: https://github.com/user-attachments/assets/af3431bb-6436-485b-b354-10828692803d - :target: https://www.webshare.io/?referral_code=eb53spijl2ih - :alt: Webshare — reliable proxies built for developers and scrapers. -.. _`Click here to try Webshare`: https://www.webshare.io/?referral_code=eb53spijl2ih -.. _`Start Scraping`: https://www.webshare.io/?referral_code=eb53spijl2ih - Stay private, fast, and fully in control ---------------------------------------- `Click here to explore BestProxy`_, your go-to solution for premium residential proxies. BestProxy's proxies ensure smooth browsing, fast speeds, and total anonymity. `Get Started`_ today and experience the difference!