This section of the docs shows how to do some useful but advanced things with newspaper.
Downloading articles one at a time is slow. But spamming a single news source like cnn.com with tons of threads or with ASYNC-IO will cause rate limiting and also doing that is very mean.
We solve this problem by allocating 1-2 threads per news source to both greatly speed up the download time while being respectful.
>>> import newspaper
>>> from newspaper import news_pool
>>> slate_paper = newspaper.build('http://slate.com')
>>> tc_paper = newspaper.build('http://techcrunch.com')
>>> espn_paper = newspaper.build('http://espn.com')
>>> papers = [slate_paper, tc_paper, espn_paper]
>>> news_pool.set(papers, threads_per_source=2) # (3*2) = 6 threads total
>>> news_pool.join()
At this point, you can safely assume that download() has been
called on every single article for all 3 sources.
>>> print slate_paper.articles[10].html
u'<html> ...'Instead of using the newspaper.build(..) api, we can take one step lower
into newspaper's Source api.
>>> from newspaper import Source
>>> cnn_paper = Source('http://cnn.com')
>>> print cnn_paper.size() # no articles, we have not built the source
0
>>> cnn_paper.build()
>>> print cnn_paper.size()
3100Note the build() method above. You may go lower level and de-abstract it
for absolute control over how your sources are constructed.
>>> cnn_paper = Source('http://cnn.com')
>>> cnn_paper.download()
>>> cnn_paper.parse()
>>> cnn_paper.set_categories()
>>> cnn_paper.download_categories()
>>> cnn_paper.parse_categories()
>>> cnn_paper.set_feeds()
>>> cnn_paper.download_feeds()
>>> cnn_paper.generate_articles()
>>> print cnn_paper.size()
3100And voila, we have mimic'd the build() method. In the above sequence,
every method is dependant on the method above it. Stop whenever you wish.
Newspaper provides two api's for users to configure their Article and
Source objects. One is via named parameter passing recommended and
the other is via Config objects.
Here are some named parameter passing examples:
>>> import newspaper
>>> from newspaper import Article, Source
>>> cnn = newspaper.build('http://cnn.com', language='en', memoize_articles=False)
>>> article = Article(url='http://cnn.com/french/...', language='fr', fetch_images=False)
>>> cnn = Source(url='http://latino.cnn.com/...', language='es', request_timeout=10,
number_threads=20)Here are some examples of how Config objects are passed.
>>> import newspaper
>>> from newspaper import Config, Article, Source
>>> config = Config()
>>> config.memoize_articles = False
>>> cbs_paper = newspaper.build('http://cbs.com', config)
>>> article_1 = Article(url='http://espn/2013/09/...', config)
>>> cbs_paper = Source('http://cbs.com', config)Here is a full list of the configuration options:
MIN_WORD_COUNT default 300 "num of word tokens in article text"
MIN_SENT_COUNT default 7 "num of sentence tokens"
MAX_TITLE default 200 "num of chars in article title"
MAX_TEXT default 100000 "num of chars in article text"
MAX_KEYWORDS default 35 "num of keywords in article"
MAX_AUTHORS default 10 "num of author names in article"
MAX_SUMMARY default 5000 "num of chars of the summary"
MAX_FILE_MEMO default 20000 "python setup.py sdist bdist_wininst upload"
parser_class default 'lxml' "lxml vs soup"
memoize_articles default True "cache and save articles run after run"
fetch_images default True "set this to false if you don't care about getting images"
language default 'en' "run newspaper.languages() to see available options."
browser_user_agent default 'newspaper/%s' % __version__
request_timeout default 7
number_threads default 10 "number of threads when mthreading"
verbose default False "turn this on when debugging"
You may notice other config options in the newspaper/configuration.py file,
however, they are private, please do not toggle them.
TODO
Here, we will define exactly how newspaper handles a lot of the data extraction.
TODO