Metatags on Vice.com

[0x0ece]

Issue, e.g., with this url (I'm using the 2.x branch):
http://motherboard.vice.com/read/when-will-humans-live-on-mars

In [2]: a = newspaper.Article(url=url)
In [3]: a.download()
In [4]: a.parse()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-03fc2775181a> in <module>()
----> 1 a.parse()

/Users/ec/devel/newspaper/newspaper/article.pyc in parse(self)
    201         self.set_meta_keywords(meta_keywords)
    202
--> 203         meta_data = self.extractor.get_meta_data(self.clean_doc)
    204         self.set_meta_data(meta_data)
    205

/Users/ec/devel/newspaper/newspaper/extractors.pyc in get_meta_data(self, doc)
    355             for idx, part in enumerate(key):
    356                 if idx == len(key) - 1:
--> 357                     ref[part] = value
    358                     break
    359                 if not ref.get(part):

TypeError: 'str' object does not support item assignment

[codelucas]

Great find! The bug is fixed on the py2 branch, soon to be fixed on master.

The issue arose from the fact that our meta tag => dictionary conversion code did not account for parent meta tags like og: and twtter: having string values themselves. The vice article you listed makes for a good example.

Here is the new output:

Python 2.7.3 (default, Sep 26 2013, 20:03:06) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from newspaper import Article
>>> a = Article("http://motherboard.vice.com/read/when-will-humans-live-on-mars")
>>> a.download()
>>> a.parse()
>>> a.meta_data
defaultdict(<type 'dict'>, {'description': u"\u200bIt's a bit of a strange thing to think about, but the Earth is going to die someday.", 'msapplication-wide310x150logo': '/assets/favicons/motherboard/mstile-310x150.png?v20141216150758', 'twitter': {'card': 'summary', 'twitter': 'When Will Humans Live on Mars?', 'description': u"\u200bIt's a bit of a strange thing to think about, but the Earth is going to die someday.", 'image': '//motherboard-images.vice.com/content-images/article/16486/1414683783836817.jpg?crop=0.5703125xw:1xh;*,*&resize=500:*&output-format=jpeg&output-quality=90', 'site': '@motherboard'}, 'msapplication-TileColor': '#7974bd', 'msapplication-square70x70logo': '/assets/favicons/motherboard/mstile-70x70.png?v20141216150758', 'msapplication-square310x310logo': '/assets/favicons/motherboard/mstile-310x310.png?v20141216150758', 'og': {'site_name': 'Motherboard', 'description': u"\u200bIt's a bit of a strange thing to think about, but the Earth is going to die someday.", 'title': 'When Will Humans Live on Mars?', 'url': 'http://motherboard.vice.com/read/when-will-humans-live-on-mars', 'image': '//motherboard-images.vice.com/content-images/article/16486/1414683783836817.jpg?crop=0.5703125xw:1xh;*,*&resize=500:*&output-format=jpeg&output-quality=90', 'type': 'article'}, 'msapplication-TileImage': '/assets/favicons/motherboard/mstile-144x144.png?v20141216150758', 'keywords': 'space, tentpoles, Mars, Mars One, moon express, made in space, space colonization, elon musk, space settlement, Mars settlement, documentaries, spaced out, documentary, video, nasa, Virgin Galactic, 3D Printing, mining', 'msapplication-square150x150logo': '/assets/favicons/motherboard/mstile-150x150.png?v20141216150758', 'thumbnail': '//motherboard-images.vice.com/content-images/article/16486/1414683783836817.jpg?crop=0.5703125xw:1xh;*,*&resize=500:*&output-format=jpeg&output-quality=90', 'viewport': 'width=device-width,initial-scale=1'})
>>> 
>>> print a.meta_data['twitter']
{'card': 'summary', 'twitter': 'When Will Humans Live on Mars?', 'description': u"\u200bIt's a bit of a strange thing to think about, but the Earth is going to die someday.", 'image': '//motherboard-images.vice.com/content-images/article/16486/1414683783836817.jpg?crop=0.5703125xw:1xh;*,*&resize=500:*&output-format=jpeg&output-quality=90', 'site': '@motherboard'}