Text Article Detection

CONTRIBUTORS.md

@@ -4,12 +4,7 @@ Lucas Ou-Yang -- lucasyangpersonal@gmail.com
 
 Thanks to the following contributors:
 -------------------------------------
-- Alex Kessinger - https://github.com/voidfiles
-- Oleg Temnov - https://github.com/otemnov
-- Matthew Ward - https://github.com/WheresWardy
-- Juliano Fischer - https://github.com/julianofischer
-- Sandeep Singh - https://github.com/techaddict
-- Michael Hood - https://github.com/michaelhood
+https://github.com/codelucas/newspaper/graphs/contributors
 
 Newspaper relied on some code of a few other open source projects:
 ------------------------------------------------------------------

README.rst

@@ -15,6 +15,8 @@ Inspired by `requests`_ for its simplicity and powered by `lxml`_ for its speed:
 .. _`tweeted by`: https://twitter.com/kennethreitz/status/419520678862548992
 .. _`The Changelog`: http://thechangelog.com/newspaper-delivers-instapaper-style-article-extraction/
 
+Basic Demo: http://newspaper-demo.herokuapp.com
+
 **We support 10+ languages and everything is in unicode!**
 
 .. code-block:: pycon
@@ -160,7 +162,6 @@ If you are certain that an *entire* news source is in one language, **go ahead a
     两年双免0手续0利率 科鲁兹掀背金融轻松购_武汉车市_武汉汽
     车网_新浪汽车_新浪网
 
-
 Documentation
 -------------
 
@@ -188,67 +189,73 @@ Get it now
 Installing newspaper is simple with `pip <http://www.pip-installer.org/>`_.
 However, you will run into fixable issues if you are trying to install on ubuntu.
 
-**If you are on ubuntu**, install using the following:
+**If you are on Debian / Ubuntu**, install using the following:
 
-::
+- Python development version, needed for Python.h::
+
+    $ sudo apt-get install python-dev
 
-    # Pre-req's for lxml
-    $ apt-get install libxml2-dev libxslt-dev
+- lxml requirements::
+
+    $ sudo apt-get install libxml2-dev libxslt-dev
+
+- For PIL to recognize .jpg images::
 
-    # For PIL to recognize .jpg
     $ sudo apt-get install libjpeg-dev zlib1g-dev libpng12-dev  
 
-    $ easy_install lxml # NOT PIP
+- Install the distribution via pip::
 
     $ pip install newspaper 
 
-    $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7
+- Download NLP related corpora::
 
+    $ curl https://raw.githubusercontent.com/codelucas/newspaper/master/download_corpora.py | python2.7
 
-**If you are on OSX**, install using the following:
 
-::
+**If you are on OSX**, install using the following, you may use both homebrew or macports:
 
-    # Pre-req's for lxml
-    $ brew install libxml2 libxslt # or the equiv command in macports
+::
 
-    $ pip install lxml
+    $ brew install libxml2 libxslt
 
-    # For PIL to recognize .jpg
-    $ brew install libtiff libjpeg webp little-cms2 # or the equiv with macports
+    $ brew install libtiff libjpeg webp little-cms2
 
     $ pip install newspaper 
 
-    $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7
+    $ curl https://raw.githubusercontent.com/codelucas/newspaper/master/download_corpora.py | python2.7
 
 
-**If you are neither using ubuntu nor mac**, install with the following:
+**Otherwise**, install with the following:
 
-::
+NOTE: You will still most likely need to install the following libraries via your package manager
 
-    # You will most likely need to install the following libraries via your
-    # package manager
-    # for lxml: libxml2-dev libxslt-dev
-    # for PIL: libjpeg-dev zlib1g-dev libpng12-dev  
+- PIL: ``libjpeg-dev`` ``zlib1g-dev`` ``libpng12-dev``
+- lxml: ``libxml2-dev`` ``libxslt-dev``
+- Python Development version: ``python-dev``
 
-    $ pip install newspaper
+::
 
-    $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7
+    $ pip install newspaper
 
+    $ curl https://raw.githubusercontent.com/codelucas/newspaper/master/download_corpora.py | python2.7
 
-It is also important to note that the line 
+Development
+-----------
 
-::
+If you'd like to contribute and hack on the newspaper project, feel free to clone
+a development version of this repository locally::
 
-    $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7
+    git clone git://github.com/codelucas/newspaper.git
 
+Once you have a copy of the source, you can embed it in your Python package,
+or install it into your site-packages easily::
 
-is not needed unless you need the natural language, ``nlp()``, features like keywords 
-extraction and summarization.
+    $ pip install -r requirements.txt
+    $ python setup.py install
 
-If you are using **ubuntu** and are still running into gcc compile errors when installing lxml, try installing
-``libxslt1-dev`` instead of ``libxslt-dev``.
+Feel free to give our testing suite a shot::
 
+    $ python tests/unit_tests.py
 
 Related Projects
 ----------------
@@ -261,12 +268,6 @@ Related Projects
 .. _`ruby-readability`: https://github.com/cantino/ruby-readability 
 .. _`java-boilerpipe`: http://boilerpipe-web.appspot.com/
 
-Todo List
----------
-
-- Add a "follow_robots.txt" option in the config object.
-- Bake in the CSSSelect and BeautifulSoup dependencies
-
 .. _`Quickstart guide`: https://newspaper.readthedocs.org/en/latest/
 .. _`The Documentation`: http://newspaper.readthedocs.org
 .. _`lxml`: http://lxml.de/

docs/index.rst

@@ -210,4 +210,3 @@ to talk about the future of this library and news extraction in general!
 .. _`email & contact me`: mailto:lucasyangpersonal@gmail.com
 .. _`python-goose's`: https://github.com/grangier/python-goose
 .. _`here`: https://github.com/codelucas/newspaper/blob/master/GOOSE-LICENSE.txt 
-

docs/user_guide/contributors.rst

@@ -9,12 +9,7 @@ Lucas Ou-Yang -- http://codelucas.com, lucasyangpersonal@gmail.com
 
 Thanks to the following contributors:
 -------------------------------------
-- Alex Kessinger - https://github.com/voidfiles
-- Oleg Temnov - https://github.com/otemnov
-- Matthew Ward - https://github.com/WheresWardy  
-- Sandeep Singh - https://github.com/techaddict
-- Michael Hood - https://github.com/michaelhood
-- Juliano Fischer - https://github.com/julianofischer
+https://github.com/codelucas/newspaper/graphs/contributors
 
 Newspaper relied on some code of a few other open source projects:
 ------------------------------------------------------------------

docs/user_guide/install.rst

@@ -8,70 +8,58 @@ The first step to using any software package is getting it properly installed.
 
 Distribute & Pip
 ----------------
-
 Installing newspaper is simple with `pip <http://www.pip-installer.org/>`_.
 However, you will run into fixable issues if you are trying to install on ubuntu.
 
+**If you are on Debian / Ubuntu**, install using the following:
 
-**If you are on ubuntu**, install using the following:
+- Python development version, needed for Python.h::
 
-::
+    $ sudo apt-get install python-dev
+
+- lxml requirements::
+
+    $ sudo apt-get install libxml2-dev libxslt-dev
 
-    # Pre-req's for lxml
-    $ apt-get install libxml2-dev libxslt-dev
+- For PIL to recognize .jpg images::
 
-    # For PIL to recognize .jpg
     $ sudo apt-get install libjpeg-dev zlib1g-dev libpng12-dev  
 
-    $ easy_install lxml # NOT PIP
+- Install the distribution via pip::
 
     $ pip install newspaper 
 
-    $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7
+- Download NLP related corpora::
 
+    $ curl https://raw.githubusercontent.com/codelucas/newspaper/master/download_corpora.py | python2.7
 
-**If you are on OSX**, install using the following:
 
-::
+**If you are on OSX**, install using the following, you may use both homebrew or macports:
 
-    # Pre-req's for lxml
-    $ brew install libxml2 libxslt # or the equiv command in macports
+::
 
-    $ pip install lxml
+    $ brew install libxml2 libxslt
 
-    # For PIL to recognize .jpg
-    $ brew install libtiff libjpeg webp little-cms2 # or the equiv with macports
+    $ brew install libtiff libjpeg webp little-cms2
 
     $ pip install newspaper 
 
-    $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7
-
-
-**If you are neither using ubuntu nor mac**, install with the following:
+    $ curl https://raw.githubusercontent.com/codelucas/newspaper/master/download_corpora.py | python2.7
 
-::
-
-    # You will most likely need to install the following libraries via your
-    # package manager
-    # for lxml: libxml2-dev libxslt-dev
-    # for PIL: libjpeg-dev zlib1g-dev libpng12-dev  
-
-    $ pip install newspaper
 
-    $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7
+**Otherwise**, install with the following:
 
+NOTE: You will still most likely need to install the following libraries via your package manager
 
-It is also important to note that the line
+- PIL: ``libjpeg-dev`` ``zlib1g-dev`` ``libpng12-dev``
+- lxml: ``libxml2-dev`` ``libxslt-dev``
+- Python Development version: ``python-dev``
 
 ::
 
-    $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7
-
-
-is not needed unless you need the natural language, ``nlp()`` features like keywords extraction and summarization.
+    $ pip install newspaper
 
-If you are using ubuntu and are still running into gcc compile errors when installing lxml, try installing
-``libxslt1-dev`` instead of ``libxslt-dev``.
+    $ curl https://raw.githubusercontent.com/codelucas/newspaper/master/download_corpora.py | python2.7
 
 Get the Code
 ------------
@@ -86,5 +74,9 @@ You can clone the public repository::
 Once you have a copy of the source, you can embed it in your Python package,
 or install it into your site-packages easily::
 
+    $ pip install -r requirements.txt
     $ python setup.py install
 
+Feel free to give our testing suite a shot::
+
+    $ python tests/unit_tests.py

newspaper/extractors.py

@@ -30,6 +30,8 @@
 TITLE_REPLACEMENTS = ReplaceSequence().create(u"»").append(u"»")
 PIPE_SPLITTER = StringSplitter("\\|")
 DASH_SPLITTER = StringSplitter(" - ")
+UNDERSCORE_SPLITTER = StringSplitter("_")
+SLASH_SPLITTER = StringSplitter("/")
 ARROWS_SPLITTER = StringSplitter("»")
 COLON_SPLITTER = StringSplitter(":")
 SPACE_SPLITTER = StringSplitter(' ')
@@ -182,6 +184,15 @@ def get_title(self, doc):
             title_text = self.split_title(title_text, DASH_SPLITTER)
             used_delimeter = True
 
+        # split title with _
+        if not used_delimeter and '_' in title_text:
+            title_text = self.split_title(title_text, UNDERSCORE_SPLITTER)
+
+        # split title with /
+        if not used_delimeter and '/' in title_text:
+            title_text = self.split_title(title_text, SLASH_SPLITTER)
+            used_delimeter = True
+
         # split title with »
         if not used_delimeter and u'»' in title_text:
             title_text = self.split_title(title_text, ARROWS_SPLITTER)
@@ -832,7 +843,7 @@ def is_table_and_no_para_exist(self, e):
                 self.parser.remove(p)
 
         subParagraphs2 = self.parser.getElementsByTag(e, tag='p')
-        if len(subParagraphs2) == 0 and e.tag is not "td":
+        if len(subParagraphs2) == 0 and e.tag != "td":
             return True
         return False
 

newspaper/parsers.py

@@ -9,6 +9,7 @@
 import logging
 import lxml.etree
 import lxml.html
+import lxml.html.clean
 
 from copy import deepcopy
 

newspaper/version.py

@@ -7,5 +7,5 @@
 __license__ = 'MIT'
 __copyright__ = 'Copyright 2014, Lucas Ou-Yang'
 
-version_info = (0, 0, 7)
+version_info = (0, 0, 8)
 __version__ = ".".join(map(str, version_info))