Article.top_node == Article.clean_top_node

[adamlwgriffiths]

In Article.parse, top_node is over-written with the cleaned node.
Then Article.clean_top_node is copied from this.
Both nodes are equal. I'm not sure what the reasons are, but it prevents extraction by external tools by hiding the extracted article html.

Preferably, Article.top_node shouldn't be over-written, and existing code should be modified to use clean_top_node where required.

[adamlwgriffiths]

I've noticed that these actually differ slightly.

>>> import newspaper
>>> url = 'http://www.rockpapershotgun.com/2014/07/24/top-down-tracy-third-eye-crime/'
>>> n = newspaper.Article(url)
>>> n.download()
>>> n.parse()
>>> 'img' in ET.tostring(n.clean_top_node)
True
>>> 'img' in ET.tostring(n.top_node)
False

clean_top_node has image tags in it, despite being a deep copy of top_node.

[codelucas]

"Both nodes are equal. I'm not sure what the reasons are, but it prevents extraction by external tools by hiding the extracted article html."

The above is a wrong statement. It isn't your fault though, the code seriously needs to be refactored, it looks incredibly unclear.

Reference: https://github.com/codelucas/newspaper/blob/master/newspaper/article.py#L209

Sure, the top_node and clean_top_node are the same at the time L207 is executed and a deep copy is made but look at L209: output_formatter.get_formatted actually modifies the top_node.

This really needs to be refactored so top_node is taken as an argument so it's more clear.

Why all of this? clean_top_node was created so that the users of newspaper could perform operations like DOM searches on the "best and most important" part of the article/HTML page. top_node by itself isn't sufficient because the node would have already been manipulated heavily after parse() and tags would be missing, as you have noticed.

[adamlwgriffiths]

Thanks for the clarification, feel free to close this bug =)

As an aside (related to why I opened this bug), if I were wanting a string version of the HTML that newspaper has localised the article to, what should I use?
I've noticed that article_html can be empty, so I'm using ElementTree.tostring(article.clean_top_node), but this seems to trigger bugs in ElementTree saying that certain nodes are NoneType, so I do a try slide (which I'd rather not do).
Is this the best way at the moment?

[codelucas]

No worries @adamlwgriffiths, thanks for using newspaper!

In the case that article_html is empty that either means you specified that it should be empty via the keep_article_html parameter (default false) or that the newspaper failed to identify a top_node to begin with.

[adamlwgriffiths]

Ah that'll be it, brilliant! Cheers