Handling 404, 500, and other non-200 http response codes to prevent scraping error pages

[deanq]

Has it ever been considered that some links would respond with a non-200 response, and this module would end up scraping the contents of that error page? I'm looking at network.get_html() and if I have time, I might just PR a change to that method to accept a strict 200-only config. I'm posting this here just in case I'm missing something or if someone has handled this elegantly already. Thanks.

[deanq]

My quick work-around for this is to handle the crawl outside and use the html parameter for newspaper.Article.download(). Here's an example:

import requests
import newspaper

try:
    website_url = "http://foobar.com/article/1234"
    site = newspaper.Article(website_url)

    page = requests.get(website_url)
    page.raise_for_status()

    site.download(html=page.content)
    site.parse()

    # do something with the `site` Article object here...

except requests.HTTPError, e:
    if e.response.status_code in [404, 410, 400]:
        # do something here...
        pass

[jourmungand]

@deanq, +1

[Tjorriemorrie]

This is a horrible oversight. I recommend that download() should at least return the response object. No reason not to. Then the dev has access to it. In addition to it, the method could also use additional arguments like elasticsearch, e.g. ignore=[400,404], raise_for_error=True, raise_for_exception=True

[sysofwan]

I suggest throwing an exception (or at least have an option to throw an exception) if the status is not 200, rather than returning an empty string (this is what the current "network.py" code does). This can help users detect and handle errors instead of failing silently.

[codelucas]

@deanq: 👍
Thanks for the feedback, I'll look into submitting a PR unless Dean beats me to it