Okay, I added a public API for multithreading article downloads (while also respecting news source domains).

Instead of going news source by source and spamming each source with X threads. We spread out 1-2 threads to each desired news source and download all of their articles concurrently so it's a WIN-WIN.

Check it out in the updated readme!

>>> import newspaper
>>> from newspaper import news_pool

>>> slate_paper = newspaper.build('http://slate.com')
>>> tc_paper = newspaper.build('http://techcrunch.com')
>>> espn_paper = newspaper.build('http://espn.com')

>>> papers = [slate_paper, tc_paper, espn_paper]
>>> news_pool.set(papers, threads_per_source=2) # (3*2) = 6 threads total
>>> news_pool.join()

At this point, you can safely assume that download() has been
called on every single article for all 3 sources.

>>> print slate_paper.articles[10].html
u'<html> ...'

This is still a very rough implementation, i'm going to need a few more commits to clean this up fully. Ideally, users should be able to customize how many threads they want to allocate per news source.

I'm also aware that you can use "privoxy" to avoid rate limiting. Not sure if we need to build that in.

Multithread & gevent framework built into newspaper #4

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions