diff --git a/.gitattributes b/.gitattributes index 4cb342ff..f8c996b6 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,2 +1,2 @@ docs/* linguist-documentation -tests/* linguist-vendored +tests/** linguist-vendored diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 00000000..fb179079 --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,9 @@ +version: 2 + +build: + os: ubuntu-24.04 + tools: + python: "3.12" + +sphinx: + configuration: docs/conf.py diff --git a/.travis.yml b/.travis.yml index f5cd2e33..8af415d8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,8 +1,8 @@ language: python python: - - "3.4" - "3.5" - "3.6" + - "3.7" install: - pip install -r requirements.txt coverage coveralls - python download_corpora.py diff --git a/README.rst b/README.rst index 8908df6d..dcbee056 100644 --- a/README.rst +++ b/README.rst @@ -13,7 +13,6 @@ Newspaper3k: Article scraping & curation :target: https://coveralls.io/github/codelucas/newspaper :alt: Coverage status - Inspired by `requests`_ for its simplicity and powered by `lxml`_ for its speed: "Newspaper is an amazing python library for extracting & curating articles." @@ -132,6 +131,8 @@ If no language is specified, Newspaper will attempt to auto detect a language. >>> print(a.title) 港特首梁振英就住宅违建事件道歉 +Multi-lingual +============= If you are certain that an *entire* news source is in one language, **go ahead and use the same api :)** @@ -163,10 +164,87 @@ If you are certain that an *entire* news source is in one language, **go ahead a 两年双免0手续0利率 科鲁兹掀背金融轻松购_武汉车市_武汉汽 车网_新浪汽车_新浪网 -Documentation -------------- -Check out `The Documentation`_ for full and detailed guides using newspaper. +Scraping by topic: where do the URLs come from? +=============================================== + +``newspaper.build()`` is perfect when you know *which sites* to crawl. But the other question I get constantly is: "I want every article about **X**, across all publications — where do I get the URLs?" The answer is to search Google News for your keyword first, then feed the result links straight into newspaper for extraction. + +The easiest way to query Google News programmatically is the `Google News API`_ from `SerpApi - Search API`_ (they also cover Google Search, Google Maps, and more). The two libraries snap together in a few lines: + +.. code-block:: python + + # pip3 install google-search-results + from serpapi import GoogleSearch + from newspaper import Article + + search = GoogleSearch({ + "engine": "google_news", + "q": "electric vehicles", + "api_key": "YOUR_SERPAPI_KEY", # free plan at serpapi.com + }) + + for result in search.get_dict()["news_results"]: + article = Article(result["link"]) + article.download() + article.parse() + article.nlp() + print(article.title, "--", article.summary[:120]) + +This pattern of SerpApi for *discovery*, newspaper3k for *extraction*, is how most production news-monitoring pipelines are built, and it sidesteps writing a crawler for every source you care about. + +.. _`SerpApi - Search API`: https://serpapi.com?utm_source=newspaper3k_github +.. _`Google News API`: https://serpapi.com/google-news-api?utm_source=newspaper3k_github + + +Scraping at scale: avoiding IP blocks +===================================== + +Once you move past scraping a handful of articles, you'll hit the same wall every news scraper hits: 403s, captchas, rate limits, and silent shadow bans. Your code is fine — your IP is the problem. The fix is rotating residential proxies. + +I personally route my own newspaper3k pipelines through `Swiftproxy`_ — 80M+ residential IPs across 195+ countries, a 99.89% success rate, non-expiring traffic, and a free trial so you can pressure-test it before paying. Plugging it into newspaper3k takes about four lines: + +.. code-block:: python + + from newspaper import Article, Config + + config = Config() + config.proxies = { + 'http': 'http://USERNAME:PASSWORD@gate.swiftproxy.net:7777', + 'https': 'http://USERNAME:PASSWORD@gate.swiftproxy.net:7777', + } + # a real browser UA helps too + config.browser_user_agent = ( + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' + 'AppleWebKit/537.36 (KHTML, like Gecko) ' + 'Chrome/124.0.0.0 Safari/537.36' + ) + config.request_timeout = 20 + + article = Article('https://example.com/some-news-story', config=config) + article.download() + article.parse() + print(article.title) + +The same ``config`` object works with ``newspaper.build()`` — every article fetched by the source will rotate through residential IPs automatically: + +.. code-block:: python + + import newspaper + paper = newspaper.build('http://cnn.com', config=config, memoize_articles=False) + for article in paper.articles: + article.download() + article.parse() + +Grab credentials and a free trial at `swiftproxy.net `_. Use code ``PROXY90`` for 10% off your first plan. + +.. _`Swiftproxy`: https://www.swiftproxy.net/?ref=codelucas + + +Docs +---- + +Check out `The Docs`_ for full and detailed guides using newspaper. Interested in adding a new language for us? Refer to: `Docs - Adding new languages `_ @@ -231,7 +309,6 @@ Features vi Vietnamese zh Chinese - Get it now ---------- @@ -298,25 +375,6 @@ NOTE: You will still most likely need to install the following libraries via you $ curl https://raw.githubusercontent.com/codelucas/newspaper/master/download_corpora.py | python3 -Consulting ----------- - -*This service is already used around the world* by startups, top news organizations (CNN, NYT, etc), -graduate school researchers, and, of course, hackers like you :) If you or your company are interested -in more advanced features like: increased NLP and scraping accuracy, mis-information, fake news, author -credibility, boosted coverage and accuracy for your use case, and etc; feel free to `email & contact me`_ -for consulting. - -Donations ---------- - -Your donations are greatly appreciated! They will free me up to work on this project more, -to take on things like: adding new features, bug-fix support, addressing concerns with the library. - -- My PayPal link: `https://www.paypal.me/codelucas`_ -- My `Venmo`_ handle: @Lucas-Ou-Yang - - Development ----------- @@ -339,7 +397,6 @@ Planning on tweaking our full-text algorithm? Add the ``fulltext`` parameter:: $ python3 tests/unit_tests.py fulltext - Demo ---- @@ -347,6 +404,44 @@ View a working online demo here: http://newspaper-demo.herokuapp.com This is another working online demo: http://newspaper.chinazt.cc/ + +Interested in scraping APIs & proxies? +====================================== + +Unlock the Web — the Smart Way +------------------------------ +`Click here to see SerpApi, scrape search engines easily with SerpApi - Search API`_. +Scrape Google Search, Google News, Google Maps, and more! + +.. image:: https://github.com/user-attachments/assets/9a80eeb4-72a8-43f1-9413-93c7a47b2bf6 + :target: https://serpapi.com/google-news-api?utm_source=newspaper3k_github + :alt: Scrape search engines easily with SerpApi - Search API. + +.. _`Click here to see SerpApi, scrape search engines easily with SerpApi - Search API`: https://serpapi.com?utm_source=newspaper3k_github + + +Power your scraping and automation at real-world scale +------------------------------------------------------ +`Click here to try Swiftproxy`_ — built for developers running scraping, automation, and data collection workflows at scale. Access 80M+ residential IPs from $0.7/GB, fast ISP proxies from $6/IP, global coverage across 195+ countries, non-expiring traffic, and a 99.89% success rate. Free trial available — use code ``PROXY90`` for 10% off. + +.. image:: https://github.com/user-attachments/assets/913f1fd6-20e9-4f37-89b7-ba6b0bd0724a + :target: https://www.swiftproxy.net/?ref=codelucas + :alt: Swiftproxy — residential and ISP proxies built for scrapers and developers. + +.. _`Click here to try Swiftproxy`: https://www.swiftproxy.net/?ref=codelucas + + +Stay private, fast, and fully in control +---------------------------------------- +`Click here to explore BestProxy`_, your go-to solution for premium residential proxies. BestProxy's proxies ensure smooth browsing, fast speeds, and total anonymity. `Get Started`_ today and experience the difference! + +.. image:: https://github.com/user-attachments/assets/1c6ef38c-f0c0-4db0-aad2-3ed9d6adf0b5 + :target: https://bestproxy.com/?keyword=b2vgzl0r + :alt: Experience BestProxy, smooth browsing, fast speeds, and total anonymity. + +.. _`Click here to explore BestProxy`: https://bestproxy.com/?keyword=b2vgzl0r +.. _`Get Started`: https://bestproxy.com/?keyword=b2vgzl0r + LICENSE ------- @@ -369,7 +464,8 @@ to talk about the future of this library and news extraction in general! .. _`Venmo`: https://www.venmo.com/Lucas-Ou-Yang .. _`Quickstart guide`: https://newspaper.readthedocs.io/en/latest/ -.. _`The Documentation`: https://newspaper.readthedocs.io +.. _`The Docs`: https://newspaper.readthedocs.io .. _`lxml`: http://lxml.de/ .. _`requests`: https://github.com/kennethreitz/requests .. _`Parse.ly`: http://parse.ly +.. _`It takes only one click`: https://tracking.gitads.io/?campaign=gitads&repo=newspaper&redirect=gitads.io diff --git a/docs/index.rst b/docs/index.rst index 111ae633..b4e0bae7 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -352,6 +352,21 @@ Newspaper uses a lot of `python-goose's`_ parsing code. View their license `here Please feel free to `email & contact me`_ if you run into issues or just would like to talk about the future of this library and news extraction in general! +Sponsored by SerpApi +-------------------- + +`Scrape search engines easily with SerpApi - Search API`_. +Scrape Google Search, Google News, Google Maps, and more! Their `Google News API`_ +pairs perfectly with newspaper: use it to discover article URLs by keyword, then +extract them with ``Article``. + +.. image:: https://github.com/user-attachments/assets/9a80eeb4-72a8-43f1-9413-93c7a47b2bf6 + :target: https://serpapi.com/google-news-api?utm_source=newspaper3k_docs + :alt: Scrape search engines easily with SerpApi - Search API. + +.. _`Scrape search engines easily with SerpApi - Search API`: https://serpapi.com?utm_source=newspaper3k_docs +.. _`Google News API`: https://serpapi.com/google-news-api?utm_source=newspaper3k_docs + .. _`Lucas Ou-Yang`: http://codelucas.com .. _`email & contact me`: mailto:lucasyangpersonal@gmail.com .. _`python-goose's`: https://github.com/grangier/python-goose diff --git a/newspaper/version.py b/newspaper/version.py index f89d0d3e..e2eab31a 100644 --- a/newspaper/version.py +++ b/newspaper/version.py @@ -7,5 +7,5 @@ __license__ = 'MIT' __copyright__ = 'Copyright 2014, Lucas Ou-Yang' -version_info = (0, 2, 8) +version_info = (0, 3, 0) __version__ = ".".join(map(str, version_info)) diff --git a/requirements.txt b/requirements.txt index 9d5c744e..61974601 100644 --- a/requirements.txt +++ b/requirements.txt @@ -11,4 +11,4 @@ python-dateutil>=2.5.3 PyYAML>=3.11 requests>=2.10.0 tinysegmenter==0.3 # TODO(codelucas): Investigate making this >=0.3 -tldextract>=2.0.1 +tldextract>=2.0.1 \ No newline at end of file diff --git a/setup.py b/setup.py index 91dc24fd..5569c7cb 100755 --- a/setup.py +++ b/setup.py @@ -47,7 +47,7 @@ setup( name='newspaper3k', - version='0.2.8', + version='0.3.0', description='Simplified python article discovery & extraction.', long_description=readme, author='Lucas Ou-Yang', diff --git a/tests/unit_tests.py b/tests/unit_tests.py index 058a6deb..69c05adf 100644 --- a/tests/unit_tests.py +++ b/tests/unit_tests.py @@ -753,8 +753,7 @@ class TestDownloadPdf(unittest.TestCase): @print_test def test_article_pdf_ignoring(self): empty_pdf = "%PDF-" # empty PDF constant - a = Article(url='http://www.technik-medien.at/ePaper_Download/' - 'IoT4Industry+Business_2018-10-31_2018-03.pdf', + a = Article(url='https://www.adobe.com/pdf/pdfs/ISO32000-1PublicPatentLicense.pdf', ignored_content_types_defaults={"application/pdf": empty_pdf, "application/x-pdf": empty_pdf, "application/x-bzpdf": empty_pdf,