diff --git a/.gitattributes b/.gitattributes index 4cb342ff..f8c996b6 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,2 +1,2 @@ docs/* linguist-documentation -tests/* linguist-vendored +tests/** linguist-vendored diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 00000000..fb179079 --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,9 @@ +version: 2 + +build: + os: ubuntu-24.04 + tools: + python: "3.12" + +sphinx: + configuration: docs/conf.py diff --git a/README.rst b/README.rst index 21373ba7..e5ec7ee7 100644 --- a/README.rst +++ b/README.rst @@ -13,7 +13,6 @@ Newspaper3k: Article scraping & curation :target: https://coveralls.io/github/codelucas/newspaper :alt: Coverage status - Inspired by `requests`_ for its simplicity and powered by `lxml`_ for its speed: "Newspaper is an amazing python library for extracting & curating articles." @@ -132,6 +131,8 @@ If no language is specified, Newspaper will attempt to auto detect a language. >>> print(a.title) 港特首梁振英就住宅违建事件道歉 +Multi-lingual +============= If you are certain that an *entire* news source is in one language, **go ahead and use the same api :)** @@ -163,9 +164,128 @@ If you are certain that an *entire* news source is in one language, **go ahead a 两年双免0手续0利率 科鲁兹掀背金融轻松购_武汉车市_武汉汽 车网_新浪汽车_新浪网 -Support our library -------------------- -`It takes only one click`_ + +Scraping by topic: where do the URLs come from? +=============================================== + +``newspaper.build()`` is perfect when you know *which sites* to crawl. But the other question I get constantly is: "I want every article about **X**, across all publications — where do I get the URLs?" The answer is to search Google News for your keyword first, then feed the result links straight into newspaper for extraction. + +The easiest way to query Google News programmatically is the `Google News API`_ from `SerpApi - Search API`_ (they also cover Google Search, Google Maps, and more). The two libraries snap together in a few lines: + +.. code-block:: python + + # pip3 install google-search-results + from serpapi import GoogleSearch + from newspaper import Article + + search = GoogleSearch({ + "engine": "google_news", + "q": "electric vehicles", + "api_key": "YOUR_SERPAPI_KEY", # free plan at serpapi.com + }) + + for result in search.get_dict()["news_results"]: + article = Article(result["link"]) + article.download() + article.parse() + article.nlp() + print(article.title, "--", article.summary[:120]) + +This pattern of SerpApi for *discovery*, newspaper3k for *extraction*, is how most production news-monitoring pipelines are built, and it sidesteps writing a crawler for every source you care about. + +.. _`SerpApi - Search API`: https://serpapi.com?utm_source=newspaper3k_github +.. _`Google News API`: https://serpapi.com/google-news-api?utm_source=newspaper3k_github + + +When ``download()`` returns a wall instead of an article +======================================================== + +A fair number of publishers now render their body copy client-side or sit behind an anti-bot check, so ``article.download()`` comes back holding a challenge page and ``article.text`` ends up empty. Two things fix the large majority of those cases: make the request from a residential IP, and let something else execute the page's JavaScript before newspaper parses it. + +`Novada`_ covers both, and neither changes how you use the library. Their residential pool is just the normal ``config.proxies`` route: + +.. code-block:: python + + from newspaper import Article, Config + + config = Config() + config.proxies = { + 'http': 'http://USERNAME-zone-res:PASSWORD@super.novada.pro:7777', + 'https': 'http://USERNAME-zone-res:PASSWORD@super.novada.pro:7777', + } + + article = Article('https://example.com/some-news-story', config=config) + article.download() + article.parse() + +For the JS-heavy or aggressively protected sources, their Web Unblocker hands back rendered HTML, which you pass to ``download(input_html=...)``. Newspaper never makes the request itself, so everything downstream — parsing, ``nlp()``, images, dates — is unchanged: + +.. code-block:: python + + import requests + from newspaper import Article + + url = 'https://example.com/some-news-story' + + html = requests.post( + 'https://webunlocker.novada.com/request', + headers={'Authorization': 'Bearer YOUR_NOVADA_KEY'}, + data={'target_url': url, 'response_format': 'html', 'js_render': 'True'}, + ).text + + article = Article(url) + article.download(input_html=html) + article.parse() + print(article.title, article.publish_date, len(article.text)) + +If all you want is the body text, ``newspaper.fulltext(html)`` takes the same HTML. Novada's pool is 100M+ residential IPs across 195+ countries, and the $15 free trial spans every product, so it costs nothing to find out whether a source that keeps failing on you is genuinely unreachable or just picky about who's asking. + +.. _`Novada`: https://www.novada.com/?github-newspaper + + +Scraping at scale: avoiding IP blocks +===================================== + +Once you move past scraping a handful of articles, you'll hit the same wall every news scraper hits: 403s, captchas, rate limits, and silent shadow bans. Your code is fine — your IP is the problem. The fix is rotating residential proxies. + +I personally route my own newspaper3k pipelines through `Swiftproxy`_ — 80M+ residential IPs across 195+ countries, a 99.89% success rate, non-expiring traffic, and a free trial so you can pressure-test it before paying. Plugging it into newspaper3k takes about four lines: + +.. code-block:: python + + from newspaper import Article, Config + + config = Config() + config.proxies = { + 'http': 'http://USERNAME:PASSWORD@gate.swiftproxy.net:7777', + 'https': 'http://USERNAME:PASSWORD@gate.swiftproxy.net:7777', + } + # a real browser UA helps too + config.browser_user_agent = ( + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' + 'AppleWebKit/537.36 (KHTML, like Gecko) ' + 'Chrome/124.0.0.0 Safari/537.36' + ) + config.request_timeout = 20 + + article = Article('https://example.com/some-news-story', config=config) + article.download() + article.parse() + print(article.title) + +The same ``config`` object works with ``newspaper.build()`` — every article fetched by the source will rotate through residential IPs automatically: + +.. code-block:: python + + import newspaper + paper = newspaper.build('http://cnn.com', config=config, memoize_articles=False) + for article in paper.articles: + article.download() + article.parse() + +Grab credentials and a free trial at `swiftproxy.net `_. Use code ``PROXY90`` for 10% off your first plan. + +.. _`Swiftproxy`: https://www.swiftproxy.net/?ref=codelucas + Docs ---- @@ -235,7 +355,6 @@ Features vi Vietnamese zh Chinese - Get it now ---------- @@ -302,15 +421,6 @@ NOTE: You will still most likely need to install the following libraries via you $ curl https://raw.githubusercontent.com/codelucas/newspaper/master/download_corpora.py | python3 -Donations ---------- - -Your donations are greatly appreciated! They will free me up to work on this project more, -to take on things like: adding new features, bug-fix support, addressing concerns with the library. - -- My PayPal link: `https://www.paypal.me/codelucas`_ -- My `Venmo`_ handle: @Lucas-Ou-Yang - Development ----------- @@ -333,7 +443,6 @@ Planning on tweaking our full-text algorithm? Add the ``fulltext`` parameter:: $ python3 tests/unit_tests.py fulltext - Demo ---- @@ -341,6 +450,55 @@ View a working online demo here: http://newspaper-demo.herokuapp.com This is another working online demo: http://newspaper.chinazt.cc/ + +Interested in scraping APIs & proxies? +====================================== + +The all-in-one solution for global data scraping +------------------------------------------------- +`Click here to get started with Novada`_ — real residential proxies and scraping solutions that give reliable access to news pages at scale. 100M+ residential IPs across 195+ countries with a 99.99% success rate, plus Web Unblocker, Scraper API, and Browser API for when you'd rather be handed clean HTML or JSON than fight a captcha. Fewer blocks, and output that drops straight into your article extraction pipeline. Start with a $15 free trial across all products. + +.. image:: https://github.com/user-attachments/assets/16a62312-f077-42ed-a042-2906074435ce + :target: https://www.novada.com/?github-newspaper + :alt: Novada — real residential proxies and scraping solutions for global data collection. + +.. _`Click here to get started with Novada`: https://www.novada.com/?github-newspaper + + +Unlock the Web — the Smart Way +------------------------------ +`Click here to see SerpApi, scrape search engines easily with SerpApi - Search API`_. +Scrape Google Search, Google News, Google Maps, and more! + +.. image:: https://github.com/user-attachments/assets/9a80eeb4-72a8-43f1-9413-93c7a47b2bf6 + :target: https://serpapi.com/google-news-api?utm_source=newspaper3k_github + :alt: Scrape search engines easily with SerpApi - Search API. + +.. _`Click here to see SerpApi, scrape search engines easily with SerpApi - Search API`: https://serpapi.com?utm_source=newspaper3k_github + + +Power your scraping and automation at real-world scale +------------------------------------------------------ +`Click here to try Swiftproxy`_ — built for developers running scraping, automation, and data collection workflows at scale. Access 80M+ residential IPs from $0.7/GB, fast ISP proxies from $6/IP, global coverage across 195+ countries, non-expiring traffic, and a 99.89% success rate. Free trial available — use code ``PROXY90`` for 10% off. + +.. image:: https://github.com/user-attachments/assets/913f1fd6-20e9-4f37-89b7-ba6b0bd0724a + :target: https://www.swiftproxy.net/?ref=codelucas + :alt: Swiftproxy — residential and ISP proxies built for scrapers and developers. + +.. _`Click here to try Swiftproxy`: https://www.swiftproxy.net/?ref=codelucas + + +Stay private, fast, and fully in control +---------------------------------------- +`Click here to explore BestProxy`_, your go-to solution for premium residential proxies. BestProxy's proxies ensure smooth browsing, fast speeds, and total anonymity. `Get Started`_ today and experience the difference! + +.. image:: https://github.com/user-attachments/assets/1c6ef38c-f0c0-4db0-aad2-3ed9d6adf0b5 + :target: https://bestproxy.com/?keyword=b2vgzl0r + :alt: Experience BestProxy, smooth browsing, fast speeds, and total anonymity. + +.. _`Click here to explore BestProxy`: https://bestproxy.com/?keyword=b2vgzl0r +.. _`Get Started`: https://bestproxy.com/?keyword=b2vgzl0r + LICENSE ------- diff --git a/docs/index.rst b/docs/index.rst index 111ae633..b4e0bae7 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -352,6 +352,21 @@ Newspaper uses a lot of `python-goose's`_ parsing code. View their license `here Please feel free to `email & contact me`_ if you run into issues or just would like to talk about the future of this library and news extraction in general! +Sponsored by SerpApi +-------------------- + +`Scrape search engines easily with SerpApi - Search API`_. +Scrape Google Search, Google News, Google Maps, and more! Their `Google News API`_ +pairs perfectly with newspaper: use it to discover article URLs by keyword, then +extract them with ``Article``. + +.. image:: https://github.com/user-attachments/assets/9a80eeb4-72a8-43f1-9413-93c7a47b2bf6 + :target: https://serpapi.com/google-news-api?utm_source=newspaper3k_docs + :alt: Scrape search engines easily with SerpApi - Search API. + +.. _`Scrape search engines easily with SerpApi - Search API`: https://serpapi.com?utm_source=newspaper3k_docs +.. _`Google News API`: https://serpapi.com/google-news-api?utm_source=newspaper3k_docs + .. _`Lucas Ou-Yang`: http://codelucas.com .. _`email & contact me`: mailto:lucasyangpersonal@gmail.com .. _`python-goose's`: https://github.com/grangier/python-goose