Skip to content

Meta tag extraction fixes - #69

Merged
codelucas merged 2 commits into
codelucas:masterfrom
karls:meta-tag-extraction-fixes
Aug 1, 2014
Merged

Meta tag extraction fixes#69
codelucas merged 2 commits into
codelucas:masterfrom
karls:meta-tag-extraction-fixes

Conversation

@karls

@karls karls commented Jul 31, 2014

Copy link
Copy Markdown
Contributor

I discovered that extracting information from the HTML was a bit broken -- at least I couldn't get it to work properly. The article.meta_data dict contained the following after parsing:

defaultdict(<type 'dict'>,
{'CPS_SITE_NAME': {},
 'application-name': {},
 'Headline': {},
 'IFS_URL': {},
 'Section': {},
 'UKFS_URL': {},
 'CPS_AUDIENCE': {},
 'msapplication-task': {},
 'Description': {},
 'CPS_ID': {},
 'CPS_PLATFORM': {},
 'msapplication-TileColor': {},
 'contentFlavor': {},
 'msapplication-window': {},
 'CPS_SECTION_PATH': {},
 'twitter': {'card': 'summary'},
 'bbcsearch_noindex': {},
 'msapplication-tooltip': {},
 'CPS_ASSET_TYPE': {},
 'msapplication-TileImage': {},
 'msapplication-starturl': {},
 'viewport': {},
 'OriginalPublicationDate': {},
 'og': {'url': 'http://www.bbc.co.uk/news/world-africa-16377824',
        'site_name': 'BBC News',
        'image': 'http://news.bbcimg.co.uk/media/images/57644000/jpg/_57644047_armed-lou-nuer-youth-in-lik.jpg',
        'type': 'article',
        'title': "'More troops' to South Sudan town"},
 'THUMBNAIL_URL': {}})

Only og, twitter etc keys were parsed correctly. This patch fixes this bug and all the keys are correctly parsed.

karls added 2 commits July 28, 2014 16:25
Meta tags were incorrectly extracted when the meta key was not in the
form of "foo:bar". The resulting value was, incorrectly, an empty dict.
codelucas added a commit that referenced this pull request Aug 1, 2014
@codelucas
codelucas merged commit e8e99bc into codelucas:master Aug 1, 2014
@codelucas

Copy link
Copy Markdown
Owner

This has been merged, nicely done.
Btw, check out my newest commit. It's a refactor on the meta data
extraction and also a more comprehensive test case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants