Added published date to the extractor+article - #63
Conversation
|
Thanks for this PR, will read & test it tomorrow. |
|
You could also parse the article HTML too using Not sure if those tags would be trimmed by newspaper's extractor though. |
|
It would be hard to identify the correct date, there's usually more than one date on a page. After digging around I found only in a small number of cases the date was neither in the meta or the URL. |
|
@SkinnyP Thanks again for this PR. After reading it over more and testing it locally my thoughts are: 1.) Reference: https://github.com/skinnyp/newspaper/blob/master/newspaper/article.py#L466 2.) Reference: https://github.com/skinnyp/newspaper/blob/master/newspaper/extractors.py#L226 However, for something as important as publishing date, we can't afford to give any incorrect value. Having a URL guesser like this can result in bogus values which are hard to debug. Please remove lines 226-232 of your Otherwise nice work! After fixing the two nits above, add 1-2 unit tests of this working and i'll be ready to accept this pull request. Thanks again! |
Added published date to the extractor+article
|
^ Oops! I just accidentally merged this commit. I then reverted that commit in a seperate commit and now this entire issue can't be re-opened. Oh well, @SkinnyP, note my above comment and just submit a new pull request with the new changes! |
Nothing fancy just checks the meta tags and then if nothing is there checks the URL for the published date.