Added support to parse dates from page body

[ivbeg]

Disclaimer: I am author of qddate lib http://github.com/ivbeg/qddate , it's designed to improve date parsing, to speed up date parsing greatly.

Right now newspaper doesn't extract dates from page text, this patch should fix it.
I've tested patch against Yandex.Zen articles https://zen.yandex.ru/media/id/5a315e1300b3dd77beb64261/kak-ustroen-nekommercheskii-sektor-v-rossii-chast-2-5a3a1e898139ba9e538c916b
They have no dates in metadata, but they have dates inside span field "21.12.2017"

Another site with dates that now could be extracted - https://www.nato.int/cps/en/natohq/news_151395.htm

And so on.

[codelucas]

Thanks so much for adding this @ivbeg. 💯 👍 We will take a look

[ilkerceng]

@ivbeg I have just tried this without waiting on new release and got this errors

article.parse()

File "C:\Users\peja\AppData\Local\Programs\Python\Python36-32\lib\site-packages\newspaper\article.py", line 243, in parse
self.clean_doc)
File "C:\Users\peja\AppData\Local\Programs\Python\Python36-32\lib\site-packages\newspaper\extractors.py", line 241, in get_publishing_date
datetime_obj = dp.parse(node.text)
File "C:\Users\peja\AppData\Local\Programs\Python\Python36-32\lib\site-packages\qddate\qdparser.py", line 138, in parse
res = self.match(text, noprefix)
File "C:\Users\peja\AppData\Local\Programs\Python\Python36-32\lib\site-packages\qddate\qdparser.py", line 100, in match
basekeys = self.__matchPrefix(text[:6])
File "C:\Users\peja\AppData\Local\Programs\Python\Python36-32\lib\site-packages\qddate\qdparser.py", line 30, in __matchPrefix
return matchPrefix(text)
File "C:\Users\peja\AppData\Local\Programs\Python\Python36-32\lib\site-packages\qddate\dirty.py", line 18, in matchPrefix
if ord(text[0].lower()) in range(ord('a'), ord('z') + 1):
TypeError: ord() expected a character, but string of length 2 found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm 2018.3.1\helpers\pydev\pydevd.py", line 1741, in
main()
File "C:\Program Files\JetBrains\PyCharm 2018.3.1\helpers\pydev\pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "C:\Program Files\JetBrains\PyCharm 2018.3.1\helpers\pydev\pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files\JetBrains\PyCharm 2018.3.1\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "D:/Ceng/saman/news/news_collector/src/basic.py", line 122, in
print(traceback.format_exc(exc))
File "C:\Users\peja\AppData\Local\Programs\Python\Python36-32\lib\traceback.py", line 163, in format_exc
return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
File "C:\Users\peja\AppData\Local\Programs\Python\Python36-32\lib\traceback.py", line 117, in format_exception
type(value), value, tb, limit=limit).format(chain=chain))
File "C:\Users\peja\AppData\Local\Programs\Python\Python36-32\lib\traceback.py", line 497, in init
capture_locals=capture_locals)
File "C:\Users\peja\AppData\Local\Programs\Python\Python36-32\lib\traceback.py", line 332, in extract
if limit >= 0:
TypeError: '>=' not supported between instances of 'TypeError' and 'int'

[ivbeg]

@ilkerceng hi, I will take a look. Could you provide article url too?

[ivbeg]

@ilkerceng I've tried a few urls and it's ok

• https://www.nato.int/cps/en/natohq/news_151395.htm
• http://government.bg/bg/prestsentar/novini/premierat-borisov-izprati-saboleznovatelna-telegrama-do-prezidenta-na-indoneziya-dzhoko-uidodo
• https://ec.europa.eu/digital-single-market/en/news/egovernment-benchmark-2018-digital-efforts-european-countries-are-visibly-paying

Test cmd line tool here
article_test.zip

I guess it could be about UTF-16 support, but I need an url or several urls to understand how to fix it.

[ilkerceng]

hi @ivbeg, i'm sorry for the latency, i will try to find which url does create the error and put the example url here.

[ilkerceng]

@ivbeg these are the some urls that gives the error.

• http://www.yenimesaj.com.tr/volkan-kocaman-ile-kendini-buldu-H1311975.htm
• http://www.hurriyet.com.tr/kampus/universiteden-anket-100-evin-49una-sanat-eseri-girdi-41091649

[ivbeg]

@ilkerceng thanks. I've found how it happens and patched. Please try latest code from http://github.com/ivbeg/qddate
It looks like that for certain letters like "İ" of turkish alphabet python returns a two char string for some_string[0] instead of character so ord(some_string[0]) fail with exception. I've added additional verification in qddate and now it should work

[ilkerceng]

hi @ivbeg, i don't know how to apply the patch, is it enough to remove qddate lib and reinstall again?

[ivbeg]

hi @ilkerceng just pull latest code from "http://github.com/ivbeg/qddate" with git pull http://github.com/ivbeg/qddate and run "python setup.py install".