Skip to content

Fulltext extraction improvement #1 - #106

Merged
codelucas merged 11 commits into
masterfrom
fulltext-improvement-1
Jan 15, 2015
Merged

Fulltext extraction improvement #1#106
codelucas merged 11 commits into
masterfrom
fulltext-improvement-1

Conversation

@codelucas

Copy link
Copy Markdown
Owner

Full-text extraction improvements from this diff:

  • <li> and <ul> list elements were usually filtered out in the old extraction mechanism, the new updated extraction code includes list elements (and even in their original form, newlines are inserted!)
  • Random paragraphs, usually heading or ending, in an article were filtered out because of a strict extraction mechanism. Changes were made to make the mechanism more lenient.

Case 1:
URL: http://lungcancer.about.com/od/Targeted-Therapy-Lung-Cancer/fl/Genetic-Testing-for-People-with-Lung-Cancer.htm?utm_source=google&utm_medium=social&utm_campaign=shareurlbuttons
Fail description: Before this PR, titles and lists <li><ul> elements were flat out ignored in the output fulltext. Also, the intro paragraph (huge) was ignored for no good reason.
Before: https://dl.dropboxusercontent.com/u/103212767/old_fulltext/about.com2.txt
After: https://dl.dropboxusercontent.com/u/103212767/new_fulltext/about.com2.txt

Case 2:
URL: http://in.reuters.com/article/2014/12/30/india-sensex-nifty-idINKBN0K80A020141230
Fail description: First paragraph of all Reuters articles are ignored for no good reason.
Before: https://dl.dropboxusercontent.com/u/103212767/old_fulltext/reuters.com1.txt
After: https://dl.dropboxusercontent.com/u/103212767/new_fulltext/reuters.com1.txt

Case 3:
URL: http://www.telegraph.co.uk/news/worldnews/asia/indonesia/11315001/AirAsia-plane-reported-missing-with-162-passengers-onboard-latest.html
Fail description: Old extractor drops random blocks of text: for example, the timestamp "07.13" is dropped and many tweets about the flight crash were also dropped (embedded tweets are becoming more commonplace within articles and should be included)
Before: https://dl.dropboxusercontent.com/u/103212767/old_fulltext/telegraph.co.uk1.txt
After: https://dl.dropboxusercontent.com/u/103212767/new_fulltext/telegraph.co.uk1.txt

@codelucas

Copy link
Copy Markdown
Owner Author

Excuse commits 4d2b68b, 2b4d2bc and e54b6ab, I included too many code changes in the commit and wanted to amend and split the commit.

@codelucas

Copy link
Copy Markdown
Owner Author

@amontalenti
NOTE: The important (full-text algorithm changing) diffs in this PR are:
7697eb0
05fd1aa

codelucas added a commit that referenced this pull request Jan 15, 2015
@codelucas
codelucas merged commit c5eefcb into master Jan 15, 2015
@codelucas
codelucas deleted the fulltext-improvement-1 branch January 15, 2015 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant