Add extraction publishing date from article.

[codelucas]

The publishing date of an article is critical. I believe a good way of extracting publishing dates is to use a set of regex patterns and/or some notable id/class names.

[thejourneydude]

You might be better off avoiding that notable id/class names and focus on the regex pattern. God knows how many times I wrote scrapers and found the HTML to be garbage.

[codelucas]

Haha thanks for the advice Jason!! You should ruby port this ;)

[skswbwt]

One issue to consider is how to deal with false positives in the page. Regex alone is not really amenable for scoring the resulting dates. Also note that dates might be relative, like "an hour ago" or "just now", which is not unlikely to appear in the article text.

One possible (maybe overcomplicated) way is to first extract some list of candidates for the publication date (with regex, or some NLP temporal tagger), build features for those candidates, then run some machine learning model on those features. Some possible features include the position relative to article title and body, certain substrings in the surrounding text or the container class/id, and whether the matched string satisfies certain regex's. There might be some tedious work getting the training data though.

[codelucas]

Thanks for the advice skswbwt! I like your idea of extracting candidates first with regex.

How to proceed from there would make for a good debate between using ML or some other strategy.

[julianofischer]

I would like to remind you that the date can be represented in a different way in accordance with the country.
(e.g. dd/mm/yyyy in Brazil whereas mm/dd/yyyy in USA, right?)

[adamramadhan]

yeah i think we should put it somewhere in language regex.

[codelucas]

This issue is resolved:
30eab17

I've tested it on 150 different articles on various domains and it has a 60% success rate. (it could be even higher, up to 90%, but at a sacrifice for accuracy. But, publishing dates are useless (and even detrimental) if not absolutely precise!

The publishing date extraction technique currently used relies on parsing dates out of the URL or metadata, which has a much higher accuracy than regexing the HTML and using heuristics to choose a proper date. This technique is also language independent!