diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index bd3d0106..c2e729f6 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -4,9 +4,12 @@ Lucas Ou-Yang -- lucasyangpersonal@gmail.com Thanks to the following contributors: ------------------------------------- -- Michael Hood - https://github.com/michaelhood -- Juliano Fischer - https://github.com/julianofischer - Alex Kessinger - https://github.com/voidfiles +- Oleg Temnov - https://github.com/otemnov +- Matthew Ward - https://github.com/WheresWardy +- Juliano Fischer - https://github.com/julianofischer +- Sandeep Singh - https://github.com/techaddict +- Michael Hood - https://github.com/michaelhood Newspaper relied on some code of a few other open source projects: ------------------------------------------------------------------ diff --git a/newspaper/packages/GOOSE_LICENSE.txt b/GOOSE-LICENSE.txt similarity index 100% rename from newspaper/packages/GOOSE_LICENSE.txt rename to GOOSE-LICENSE.txt diff --git a/HISTORY.md b/HISTORY.md deleted file mode 100644 index 3f1ecd07..00000000 --- a/HISTORY.md +++ /dev/null @@ -1,16 +0,0 @@ -0.0.4 - Fully integrated python-goose library into newspaper. Article objects - now have much more options. All configurations are now based on Configuration() - objects which can be passed into Source or Article objects. Default configuration - setups make this easy. Added simple multithreading article download framework. - -0.0.5 - Fixed seamless configuration api for Article and Source objects. Enabled multi language - support in 10+ languages including non-western languages like Arabic, Korean, Chinese. - Fixed bug where we made a wrong assumption of calling .text from the requests module. - -0.0.6 - Fixed a bunch of small bugs in the source.py file (still need to update readme). Batch - downloading articles was not setting the article is_downloaded boolean. I was also using - the del keyword very irresponsibly... Made many modifications where the source object - had to filter out urls. Mostly replaced with list comprehensions. - Added a pull request from Alex K. where he added an option for just the article html - extraction. Feel free to toggle this option in the configs. I have yet to add this to the - docs once again. diff --git a/README.rst b/README.rst index 0b2bdc2b..cf0249b1 100644 --- a/README.rst +++ b/README.rst @@ -5,7 +5,15 @@ Newspaper: Article scraping & curation :target: http://badge.fury.io/py/newspaper :alt: Latest version -*Newspaper* is a Python 2 library for extracting & curating articles from the web. It is inspired by `requests`_ for its simplicity and powered by `lxml`_ for its speed. +Inspired by `requests`_ for its simplicity and powered by `lxml`_ for its speed: + + "Newspaper is an amazing python library for extracting & curating articles." + -- `tweeted by`_ Kenneth Reitz, Author of `requests`_ + + "Newspaper delivers Instapaper style article extraction." -- `The Changelog`_ + +.. _`tweeted by`: https://twitter.com/kennethreitz/status/419520678862548992 +.. _`The Changelog`: http://thechangelog.com/newspaper-delivers-instapaper-style-article-extraction/ **We support 10+ languages and everything is in unicode!** @@ -14,10 +22,12 @@ Newspaper: Article scraping & curation >>> import newspaper >>> newspaper.languages() - Your available langauges are: + Your available languages are: input code full name ar Arabic + ru Russian + nl Dutch de German en English es Spanish @@ -25,20 +35,14 @@ Newspaper: Article scraping & curation it Italian ko Korean no Norwegian - pt Portugease + pt Portuguese sv Swedish + hu Hungarian + fi Finnish + da Danish zh Chinese - -Praise: -------- - - Newspaper is an amazing python library for extracting & curating articles. - -- `tweeted by`_ Kenneth Reitz, Author of `requests`_ - - Newspaper delivers Instapaper style article extraction. -- `The Changelog`_ - -.. _`tweeted by`: https://twitter.com/kennethreitz/status/419520678862548992 -.. _`The Changelog`: http://thechangelog.com/ + id Indonesian + vi Vietnamese A Glance: --------- @@ -131,7 +135,7 @@ If you are certain that an *entire* news source is in one language, **go ahead a .. code-block:: pycon >>> import newspaper - >>> sina_paper = newspaper.build('http://www.sina.com.cn/', langauge='zh') + >>> sina_paper = newspaper.build('http://www.sina.com.cn/', language='zh') >>> for category in sina_paper.category_urls(): >>> print category @@ -162,6 +166,8 @@ Documentation Check out `The Documentation`_ for full and detailed guides using newspaper. +Interested in adding a new language for us? Refer to: `Docs - Adding new languages `_ + Features -------- @@ -182,28 +188,54 @@ Get it now Installing newspaper is simple with `pip `_. However, you will run into fixable issues if you are trying to install on ubuntu. -**If you are not using ubuntu**, install with the following: +**If you are on ubuntu**, install using the following: :: - $ pip install newspaper + # Pre-req's for lxml + $ apt-get install libxml2-dev libxslt-dev + + # For PIL to recognize .jpg + $ sudo apt-get install libjpeg-dev zlib1g-dev libpng12-dev + + $ easy_install lxml # NOT PIP + + $ pip install newspaper $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7 -**If you are**, install using the following: +**If you are on OSX**, install using the following: :: - $ apt-get install libxml2-dev libxslt-dev + # Pre-req's for lxml + $ brew install libxml2 libxslt # or the equiv command in macports + + $ pip install lxml + + # For PIL to recognize .jpg + $ brew install libtiff libjpeg webp little-cms2 # or the equiv with macports - $ easy_install lxml # NOT PIP - $ pip install newspaper $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7 +**If you are neither using ubuntu nor mac**, install with the following: + +:: + + # You will most likely need to install the following libraries via your + # package manager + # for lxml: libxml2-dev libxslt-dev + # for PIL: libjpeg-dev zlib1g-dev libpng12-dev + + $ pip install newspaper + + $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7 + + It is also important to note that the line :: @@ -211,11 +243,24 @@ It is also important to note that the line $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7 -is not needed unless you need the natural language, ``nlp()``, features like keywords extraction and summarization. +is not needed unless you need the natural language, ``nlp()``, features like keywords +extraction and summarization. -If you are using ubuntu and are still running into gcc compile errors when installing lxml, try installing +If you are using **ubuntu** and are still running into gcc compile errors when installing lxml, try installing ``libxslt1-dev`` instead of ``libxslt-dev``. + +Related Projects +---------------- + +- `ruby-readability`_ is a port of arc90's readability project to Ruby. +- `python-goose`_ is a port of Gravity's goose project to Python. +- `java-boilerpipe`_ is an article extraction library in Java. + +.. _`python-goose`: https://github.com/grangier/python-goose +.. _`ruby-readability`: https://github.com/cantino/ruby-readability +.. _`java-boilerpipe`: http://boilerpipe-web.appspot.com/ + Todo List --------- @@ -227,3 +272,17 @@ Todo List .. _`lxml`: http://lxml.de/ .. _`requests`: https://github.com/kennethreitz/requests +LICENSE +------- + +Authored and maintained by `Lucas Ou-Yang`_. + +Newspaper uses a lot of `python-goose's`_ parsing code. View their license `here`_. + +Please feel free to `email & contact me`_ if you run into issues or just would like +to talk about the future of this library and news extraction in general! + +.. _`Lucas Ou-Yang`: http://codelucas.com +.. _`email & contact me`: mailto:lucasyangpersonal@gmail.com +.. _`python-goose's`: https://github.com/grangier/python-goose +.. _`here`: https://github.com/codelucas/newspaper/blob/master/GOOSE-LICENSE.txt diff --git a/docs/index.rst b/docs/index.rst index 8c5a9941..79c65d29 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,9 +1,17 @@ Newspaper: Article scraping & curation ====================================== -Release v0.0.5. :ref:`(Installation) `. +Release v0.0.7. :ref:`(Installation) `. -*Newspaper* is a Python 2 library for extracting & curating articles from the web. It is inspired by `requests`_ for its simplicity and powered by `lxml`_ for its speed. +Inspired by `requests`_ for its simplicity and powered by `lxml`_ for its speed. + + "Newspaper is an amazing python library for extracting & curating articles." + -- `tweeted by`_ Kenneth Reitz, Author of `requests`_ + + "Newspaper delivers Instapaper style article extraction." -- `The Changelog`_ + +.. _`tweeted by`: https://twitter.com/kennethreitz/status/419520678862548992 +.. _`The Changelog`: http://thechangelog.com/ **We support 10+ languages and everything is in unicode!** @@ -16,6 +24,8 @@ Release v0.0.5. :ref:`(Installation) `. input code full name ar Arabic + ru Russian + nl Dutch de German en English es Spanish @@ -23,10 +33,14 @@ Release v0.0.5. :ref:`(Installation) `. it Italian ko Korean no Norwegian - pt Portugease + pt Portuguese sv Swedish + hu Hungarian + fi Finnish + da Danish zh Chinese - + id Indonesian + vi Vietnamese A Glance: --------- @@ -119,7 +133,7 @@ If you are certain that an *entire* news source is in one language, **go ahead a .. code-block:: pycon >>> import newspaper - >>> sina_paper = newspaper.build('http://www.sina.com.cn/', langauge='zh') + >>> sina_paper = newspaper.build('http://www.sina.com.cn/', language='zh') >>> for category in sina_paper.category_urls(): >>> print category @@ -178,8 +192,22 @@ User Guide .. _`lxml`: http://lxml.de/ .. _`nltk`: http://nltk.org/ -.. _`requests`: http://docs.python-requests.org/en/latest/ +.. _`requests`: https://github.com/kennethreitz/requests .. _`goose`: https://github.com/grangier/python-goose +LICENSE +------- + +Authored and maintained by `Lucas Ou-Yang`_. + +Newspaper uses a lot of `python-goose's`_ parsing code. View their license `here`_. + +Please feel free to `email & contact me`_ if you run into issues or just would like +to talk about the future of this library and news extraction in general! + +.. _`Lucas Ou-Yang`: http://lucasou.com +.. _`email & contact me`: mailto:lucasyangpersonal@gmail.com +.. _`python-goose's`: https://github.com/grangier/python-goose +.. _`here`: https://github.com/codelucas/newspaper/blob/master/GOOSE-LICENSE.txt diff --git a/docs/user_guide/advanced.rst b/docs/user_guide/advanced.rst index 8c0b8de6..bdfa8714 100644 --- a/docs/user_guide/advanced.rst +++ b/docs/user_guide/advanced.rst @@ -35,6 +35,81 @@ speed up the download time while being respectful. >>> print slate_paper.articles[10].html u' ...' +Keeping Html of main body article +--------------------------------- + +Keeping the html of just an article's body text is helpbut because it allows you +to retain some of the semantic information in the html. Also it will help if you +end up displaying the extracted article somehow. + +Here is how to do so: + +.. code-block:: pycon + + >>> from newspaper import Article + + >>> a = Article('http://www.cnn.com/2014/01/12/world/asia/north-korea-charles-smith/index.html' + , keep_article_html=True) + + >>> a.download() + >>> a.parse() + + >>> a.article_html + u'
\n

(CNN) -- Charles Smith insisted Sunda...' + +The lxml (dom object) and top_node (chunk of dom that contains our 'Article') are also +cached incase users would like to use them. + +Access **after parsing()** with: + +.. code-block:: pycon + + >>> a.download() + >>> a.parse() + >>> a.clean_dom + + + >>> a.clean_top_node + + + +Adding new languages +-------------------- + +First, please reference this file and read from the highlighted line all the way +down to the end of the file. + +`https://github.com/codelucas/newspaper/blob/master/newspaper/text.py#L57 `_ + +One aspect of our text extraction algorithm revolves around counting the number of +**stopwords** present in a text. Stopwords are: *some of the most common, short +function words, such as the, is, at, which, and on* in a language. + +Reference this line to see it in action: +`https://github.com/codelucas/newspaper/blob/master/newspaper/extractors.py#L669 `_ + +**So for latin languages**, it is pretty basic. We first provide a list of +stopwords in ``stopwords-.txt`` form. We then take some input text and +tokenize it into words by splitting the white space. After that we perform some +bookkeeping and then proceed to count the number of stopwords present. + +**For non-latin languages**, as you may have noticed in the code above, we need to +tokenize the words in a different way, *splitting by whitespace simply won't work for +languages like Chinese or Arabic*. For the Chinese language we are using a whole new +open source library called *jieba* to split the text into words. For arabic we are +using a special nltk tokenizer to do the same job. + +**So, to add full text extraction to a new (non-latin) language, we need:** +#. Push up a stopwords file in the format of ``stopwords-<2-char-language-code>.txt`` +in ``newspaper/resources/text/.`` +#. Provide a way of splitting/tokenizing text in that foreign language into words. +Here are some examples for Chinese, Arabic, English: + +**For latin languages:** +#. Push up a stopwords file in the format of ``stopwords-<2-char-language-code>.txt`` +in ``newspaper/resources/text/.`` and we are done! + + Explicitly building a news source --------------------------------- @@ -115,35 +190,41 @@ Here are some examples of how Config objects are passed. Here is a full list of the configuration options: -``MIN_WORD_COUNT`` default 300 "num of word tokens in article text" +``keep_article_html``, default False, "set to True if you want to preserve html of body text" + +``MIN_WORD_COUNT``, default 300, "num of word tokens in article text" + +``MIN_SENT_COUNT``, default 7, "num of sentence tokens" + +``MAX_TITLE``, default 200, "num of chars in article title" + +``MAX_TEXT``, default 100000, "num of chars in article text" -``MIN_SENT_COUNT`` default 7 "num of sentence tokens" +``MAX_KEYWORDS``, default 35, "num of keywords in article" -``MAX_TITLE`` default 200 "num of chars in article title" +``MAX_AUTHORS``, default 10, "num of author names in article" -``MAX_TEXT`` default 100000 "num of chars in article text" +``MAX_SUMMARY``, default 5000, "num of chars of the summary" -``MAX_KEYWORDS`` default 35 "num of keywords in article" +``MAX_FILE_MEMO``, default 20000, "python setup.py sdist bdist_wininst upload" -``MAX_AUTHORS`` default 10 "num of author names in article" +``parser_class``, default 'lxml', "lxml vs soup" -``MAX_SUMMARY`` default 5000 "num of chars of the summary" +``memoize_articles``, default True, "cache and save articles run after run" -``MAX_FILE_MEMO`` default 20000 "python setup.py sdist bdist_wininst upload" +``fetch_images``, default True, "set this to false if you don't care about getting images" -``parser_class`` default 'lxml' "lxml vs soup" +``image_dimension_ration``, default 16/9.0, "max ratio for height/width, we ignore if greater" -``memoize_articles`` default True "cache and save articles run after run" +``language``, default 'en', "run ``newspaper.languages()`` to see available options." -``fetch_images`` default True "set this to false if you don't care about getting images" +``browser_user_agent``, default 'newspaper/%s' % __version__ -``language`` default 'en' "run ``newspaper.languages()`` to see available options." +``request_timeout``, default 7 -``browser_user_agent`` default 'newspaper/%s' % __version__ -``request_timeout`` default 7 -``number_threads`` default 10 "number of threads when mthreading" +``number_threads``, default 10, "number of threads when mthreading" -``verbose`` default False "turn this on when debugging" +``verbose``, default False, "turn this on when debugging" You may notice other config options in the ``newspaper/configuration.py`` file, however, they are private, **please do not toggle them**. diff --git a/docs/user_guide/contributors.rst b/docs/user_guide/contributors.rst index b1e16c31..326f5cbf 100644 --- a/docs/user_guide/contributors.rst +++ b/docs/user_guide/contributors.rst @@ -9,6 +9,10 @@ Lucas Ou-Yang -- http://codelucas.com, lucasyangpersonal@gmail.com Thanks to the following contributors: ------------------------------------- +- Alex Kessinger - https://github.com/voidfiles +- Oleg Temnov - https://github.com/otemnov +- Matthew Ward - https://github.com/WheresWardy +- Sandeep Singh - https://github.com/techaddict - Michael Hood - https://github.com/michaelhood - Juliano Fischer - https://github.com/julianofischer diff --git a/docs/user_guide/install.rst b/docs/user_guide/install.rst index 7d05d539..9ec104c3 100644 --- a/docs/user_guide/install.rst +++ b/docs/user_guide/install.rst @@ -12,27 +12,55 @@ Distribute & Pip Installing newspaper is simple with `pip `_. However, you will run into fixable issues if you are trying to install on ubuntu. -**If you are not using ubuntu**, install with the following: + +**If you are on ubuntu**, install using the following: :: - $ pip install newspaper + # Pre-req's for lxml + $ apt-get install libxml2-dev libxslt-dev + + # For PIL to recognize .jpg + $ sudo apt-get install libjpeg-dev zlib1g-dev libpng12-dev + + $ easy_install lxml # NOT PIP + + $ pip install newspaper $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7 -**If you are**, install using the following: +**If you are on OSX**, install using the following: :: - $ apt-get install libxml2-dev libxslt-dev - $ easy_install lxml # NOT PIP + # Pre-req's for lxml + $ brew install libxml2 libxslt # or the equiv command in macports + + $ pip install lxml + + # For PIL to recognize .jpg + $ brew install libtiff libjpeg webp little-cms2 # or the equiv with macports $ pip install newspaper $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7 +**If you are neither using ubuntu nor mac**, install with the following: + +:: + + # You will most likely need to install the following libraries via your + # package manager + # for lxml: libxml2-dev libxslt-dev + # for PIL: libjpeg-dev zlib1g-dev libpng12-dev + + $ pip install newspaper + + $ curl https://raw.github.com/codelucas/newspaper/master/download_corpora.py | python2.7 + + It is also important to note that the line :: diff --git a/docs/user_guide/quickstart.rst b/docs/user_guide/quickstart.rst index 81885836..c847ebbe 100644 --- a/docs/user_guide/quickstart.rst +++ b/docs/user_guide/quickstart.rst @@ -244,7 +244,7 @@ of popular news source urls.. In case you need help choosing a news source! >>> newspaper.languages() - Your available langauges are: + Your available languages are: input code full name ar Arabic @@ -255,8 +255,8 @@ of popular news source urls.. In case you need help choosing a news source! it Italian ko Korean no Norwegian - pt Portugease + pt Portuguese sv Swedish zh Chinese - \ No newline at end of file + diff --git a/newspaper/__init__.py b/newspaper/__init__.py index 83901c2f..08cc8e20 100644 --- a/newspaper/__init__.py +++ b/newspaper/__init__.py @@ -1,9 +1,7 @@ # -*- coding: utf-8 -*- - """ Wherever smart people work, doors are unlocked. -- Steve Wozniak """ - __title__ = 'newspaper' __author__ = 'Lucas Ou-Yang' __license__ = 'MIT' diff --git a/newspaper/article.py b/newspaper/article.py index abe9e292..575533a4 100644 --- a/newspaper/article.py +++ b/newspaper/article.py @@ -59,9 +59,12 @@ def __init__(self, url, title=u'', source_url=u'', config=None, **kwargs): self.title = encodeValue(title) # the url of the "best image" to represent this article, via reddit algorithm - self.top_img = u'' + self.top_img = self.top_image = u'' - self.imgs = [] # all image urls + # stores image provided by metadata + self.meta_img = u'' + + self.imgs = self.images = [] # all image urls self.movies = [] # youtube, vimeo, etc # pure text from the article @@ -101,17 +104,23 @@ def __init__(self, url, title=u'', source_url=u'', config=None, **kwargs): # meta favicon field in HTML source self.meta_favicon = u"" + # Meta tags contain a lot of structured data like OpenGraph + self.meta_data = {} + # The canonical link of this article if found in the meta data self.canonical_link = u"" # Holds the top Element we think is a candidate for the main body self.top_node = None + # Holds clean version of top Element + self.clean_top_node = None + # the lxml doc object self.doc = None # a pure object from the orig html without any cleaning options done on it - self.raw_doc = None + self.clean_doc = None # A property bucket for consumers of goose to store custom data extractions. self.additional_data = {} @@ -143,7 +152,7 @@ def parse(self): raise ArticleException() self.doc = self.parser.fromstring(self.html) - self.raw_doc = copy.deepcopy(self.doc) + self.clean_doc = copy.deepcopy(self.doc) if self.doc is None: print '[Article parse ERR] %s' % self.url @@ -180,6 +189,9 @@ def parse(self): meta_keywords = self.extractor.get_meta_keywords(self) self.set_meta_keywords(meta_keywords) + meta_data = self.extractor.get_meta_data(self) + self.set_meta_data(meta_data) + # TODO self.publish_date = self.config.publishDateExtractor.extract(self.doc) # before we do any computations on the body itself, we must clean up the document @@ -192,24 +204,36 @@ def parse(self): self.set_movies(video_extractor.get_videos()) self.top_node = self.extractor.post_cleanup(self.top_node) + self.clean_top_node = copy.deepcopy(self.top_node) + text, article_html = output_formatter.get_formatted(self) self.set_article_html(article_html) self.set_text(text) - if self.raw_doc is not None: - if self.config.fetch_images: - img_url = self.extractor.get_top_img_url(self) - self.set_top_img(img_url) - - top_imgs = self.extractor.get_img_urls(self) - self.set_imgs(top_imgs) - if self.config.fetch_images: - self.set_reddit_top_img() + self.fetch_images() self.is_parsed = True self.release_resources() + def fetch_images(self): + if self.clean_doc is not None: + meta_img_url = self.extractor.get_meta_img_url(self) + self.set_meta_img(meta_img_url) + + imgs = self.extractor.get_img_urls(self) + self.set_imgs(imgs) + + if self.clean_top_node is not None and not self.has_top_image(): + first_img = self.extractor.get_first_img_url(self) + self.set_top_img(first_img) + + if not self.has_top_image(): + self.set_reddit_top_img() + + def has_top_image(self): + return self.top_img is not None and self.top_img != u'' + def is_valid_url(self): """ Performs a check on the url of this link to @@ -225,7 +249,7 @@ def is_valid_body(self): if not self.is_parsed: raise ArticleException('must parse article before checking \ if it\'s body is valid!') - meta_type = self.parser.get_meta_type(self.raw_doc) + meta_type = self.extractor.get_meta_type(self) wordcount = self.text.split(' ') sentcount = self.text.split('.') @@ -338,16 +362,21 @@ def release_resources(self): pass # os.remove(path) - def set_reddit_top_img(self): + def set_reddit_top_img(self, test_run=False): """ Wrapper for setting images, queries known image attributes first, uses Reddit's img algorithm as a fallback. """ - if self.top_img != u'': # if we already have a top img... - return + + #todo: move tests from here + if test_run: + s = images.Scraper(self) + img = s.largest_image_url() + print 'it worked, the img is', img + try: s = images.Scraper(self) - self.set_top_img(s.largest_image_url()) + self.set_top_img_no_ckeck(s.largest_image_url()) except Exception, e: log.critical('jpeg error with PIL, %s' % e) @@ -389,7 +418,17 @@ def set_article_html(self, article_html): if article_html: self.article_html = encodeValue(article_html) + def set_meta_img(self, src_url): + self.meta_img = encodeValue(src_url) + self.set_top_img(src_url) + def set_top_img(self, src_url): + if src_url is not None: + s = images.Scraper(self) + if s.satisfies_requirements(src_url): + self.set_top_img_no_ckeck(src_url) + + def set_top_img_no_ckeck(self, src_url): """ We want to provide 2 api's for images. One at "top_img", "imgs" and one at "top_image", "images". @@ -456,6 +495,9 @@ def set_meta_description(self, meta_description): """ self.meta_description = meta_description + def set_meta_data(self, meta_data): + self.meta_data = meta_data + def set_canonical_link(self, canonical_link): """ """ diff --git a/newspaper/cleaners.py b/newspaper/cleaners.py index dd11e23c..dd1b9203 100644 --- a/newspaper/cleaners.py +++ b/newspaper/cleaners.py @@ -1,5 +1,7 @@ # -*- coding: utf-8 -*- """ +Holds the code for cleaning out unwanted tags from the lxml +dom xpath. """ from .utils import ReplaceSequence @@ -7,12 +9,14 @@ class DocumentCleaner(object): def __init__(self, config): self.config = config - # parser self.parser = self.config.get_parser() + self.remove_nodes_re = ( - "^side$|combx|retweet|mediaarticlerelated|menucontainer|navbar" + "^side$|combx|retweet|mediaarticlerelated|menucontainer|" + "navbar|storytopbar-bucket|utility-bar|inline-share-tools" "|comment|PopularQuestions|contact|foot|footer|Footer|footnote" - "|cnn_strycaptiontxt|links|meta$|scroll|shoutbox|sponsor" + "|cnn_strycaptiontxt|cnn_html_slideshow|cnn_strylftcntnt" + "|links|meta$|shoutbox|sponsor" "|tags|socialnetworking|socialNetworking|cnnStryHghLght" "|cnn_stryspcvbx|^inset$|pagetools|post-attributes" "|welcome_form|contentTools2|the_answers" @@ -41,6 +45,7 @@ def clean(self, article): """ """ doc_to_clean = article.doc + doc_to_clean = self.clean_body_classes(doc_to_clean) doc_to_clean = self.clean_article_tags(doc_to_clean) doc_to_clean = self.clean_em_tags(doc_to_clean) doc_to_clean = self.remove_drop_caps(doc_to_clean) @@ -57,6 +62,15 @@ def clean(self, article): doc_to_clean = self.div_to_para(doc_to_clean, 'span') return doc_to_clean + def clean_body_classes(self, doc): + # we don't need body classes + # in case it matches an unwanted class all the document + # will be empty + elements = self.parser.getElementsByTag(doc, tag="body") + if elements: + self.parser.delAttribute(elements[0], attr="class") + return doc + def clean_article_tags(self, doc): articles = self.parser.getElementsByTag(doc, tag='article') for article in articles: @@ -124,7 +138,7 @@ def remove_nodes_regex(self, doc, pattern): return doc def clean_para_spans(self, doc): - spans = self.parser.css_select(doc, 'p > span') + spans = self.parser.css_select(doc, 'p span') for item in spans: self.parser.drop_tag(item) return doc diff --git a/newspaper/configuration.py b/newspaper/configuration.py index 36a9d065..744f4cff 100644 --- a/newspaper/configuration.py +++ b/newspaper/configuration.py @@ -1,9 +1,14 @@ # -*- coding: utf-8 -*- """ -Config settings for both Source and Article objects. -Pass these in (optionally) via the constructors. Or -else a default Configuration() object will be used. +This class holds configuration objects, which can be thought of +as settings.py but dynamic and changing for whatever parent object +holds them. For example, pass in a config object to an Article +object, Source object, or even network methods, and it just works. """ +__title__ = 'newspaper' +__author__ = 'Lucas Ou-Yang' +__license__ = 'MIT' +__copyright__ = 'Copyright 2014, Lucas Ou-Yang' import logging @@ -12,10 +17,51 @@ from .text import StopWordsArabic from .text import StopWordsKorean from .parsers import Parser, ParserSoup +from .urls import is_abs_url, get_domain from .version import __version__ log = logging.getLogger(__name__) + +""" +class HintsDict(dict): + def __init__(self, *args): + ''' + Works like a regular dict but the :meth:`get` method is different. + + ~~Hints~~ new feature: + Example usage: + Suppose newspaper article extractor was having issues with extracting + text from 'Quartz' at qz.com. We can aid our extractor by providing simple + hints obtained by just glancing at the page, like tag names or attributes & values + of the tag containing our article. + + After inspecting a random quartz article suppose the body text is clearly + in a

tag with a class labeled 'bodyArticle'. + + {'www.qz.com': {'tag': 'div', 'attr': 'class', 'value': 'bodyArticle'} + ...} + + ''' + dict.__init__(self, args) + + def set(self, key, val): + key = get_domain(key) if is_abs_url(key) else key + if isinstance(val, dict): + self[key] = val + else: + raise Exception("Not valid Hint value, must be in the form of \ + {'www.news.domain': {'tag': 'div', 'attr': 'class', 'value': 'bodyArticle'}") + + def get(self, key, default=None): + key = get_domain(key) if is_abs_url(key) else key + try: + val = self[key] + except (KeyError, ValueError): + val = default + return val +""" + class Configuration(object): def __init__(self): @@ -41,6 +87,7 @@ def __init__(self): # set this to false if you don't care about getting images self.fetch_images = True + self.image_dimension_ration = 16/9.0 # don't toggle this variable self.use_meta_language = True @@ -63,6 +110,9 @@ def __init__(self): # set this to False if you want to recompute the categories *every* time # self.use_cached_categories = True # TODO: Make this work + # self.hints = None TODO: Maybe a future release? + + def get_language(self): return self._language diff --git a/newspaper/extractors.py b/newspaper/extractors.py index fe81619b..2ce7a462 100644 --- a/newspaper/extractors.py +++ b/newspaper/extractors.py @@ -1,11 +1,20 @@ # -*- coding: utf-8 -*- - """ +Newspaper uses a lot of python-goose's extraction code. View their +license here: https://github.com/codelucas/newspaper/blob/master/GOOSE-LICENSE.txt + +Keep all html page extraction code within this file. PLEASE abstract any +lxml or soup parsing mechanisms in the parsers.py file! """ +__title__ = 'newspaper' +__author__ = 'Lucas Ou-Yang' +__license__ = 'MIT' +__copyright__ = 'Copyright 2014, Lucas Ou-Yang' import re import copy import urlparse +from collections import defaultdict from .packages.tldextract import tldextract from .utils import ( @@ -115,7 +124,7 @@ def parse_byline(search_str): VALS = ['author', 'byline'] matches = [] _authors, authors = [], [] - doc = article.doc + doc = article.clean_doc html = article.html for attr in ATTRS: @@ -164,7 +173,7 @@ def get_title(self, article): Fetch the article title and analyze it. """ title = '' - doc = article.doc + doc = article.clean_doc title_element = self.parser.getElementsByTag(doc, tag='title') # no title found @@ -217,6 +226,29 @@ def split_title(self, title, splitter): title = title_pieces[large_text_index] return TITLE_REPLACEMENTS.replaceAll(title).strip() + def get_feed_urls(self, source_or_category): + """ + Returns list of feed urls on a source or category object. + """ + # This feels really weird..., needs more refactoring + is_source = source_or_category.__class__.__name__ == 'Source' + + if not is_source: # recursively call method with Source's categories + feed_urls = [] + for category in source_or_category.categories: + feed_urls.extend(self.get_feed_urls(category)) + feed_urls = feed_urls[:50] + feed_urls = [prepare_url(f, source.url) for f in feed_urls] + + feed_urls = list(set(feed_urls)) + return feed_urls + + doc = source_or_category.doc # it's a Category + kwargs = {'attr': 'type', 'value': 'application/rss+xml'} + feed_elements = self.parser.getElementsByTag(doc, **kwargs) + feed_urls = [e.get('href') for e in feed_elements if e.get('href')] + return feed_urls + def get_favicon(self, article): """ Extract the favicon from a website @@ -225,7 +257,7 @@ def get_favicon(self, article): """ kwargs = {'tag': 'link', 'attr': 'rel', 'value': 'icon'} - meta = self.parser.getElementsByTag(article.doc, **kwargs) + meta = self.parser.getElementsByTag(article.clean_doc, **kwargs) if meta: favicon = self.parser.getAttribute(meta[0], 'href') return favicon @@ -236,7 +268,7 @@ def get_meta_lang(self, article): Extract content language from meta. """ # we have a lang attribute in html - attr = self.parser.getAttribute(article.doc, attr='lang') + attr = self.parser.getAttribute(article.clean_doc, attr='lang') if attr is None: # look up for a Content-Language in meta items = [ @@ -244,7 +276,7 @@ def get_meta_lang(self, article): {'tag': 'meta', 'attr': 'name', 'value': 'lang'} ] for item in items: - meta = self.parser.getElementsByTag(article.doc, **item) + meta = self.parser.getElementsByTag(article.clean_doc, **item) if meta: attr = self.parser.getAttribute(meta[0], attr='content') break @@ -258,6 +290,10 @@ def get_meta_lang(self, article): def get_meta_content(self, doc, metaName): """ Extract a given meta content form document. + Example metaNames: + "meta[name=description]" + "meta[name=keywords]" + "meta[property=og:type]" """ meta = self.parser.css_select(doc, metaName) content = None @@ -270,24 +306,102 @@ def get_meta_content(self, doc, metaName): return '' - def get_meta_description(self, article): + def get_meta_img_url(self, article): + """ + Returns the 'top img' as specified by the website. + """ + top_meta_image, try_one, try_two, try_three, try_four = [None] * 5 + doc = article.clean_doc + + try_one = self.get_meta_content(doc, 'meta[property="og:image"]') + if try_one is None: + link_icon_kwargs = {'tag': 'link', 'attr': 'rel', 'value': 'icon'} + elems = self.parser.getElementsByTag(doc, **link_icon_kwargs) + try_two = elems[0].get('href') if elems else None + + if try_two is None: + link_img_src_kwargs = {'tag': 'link', 'attr': 'rel', 'value': 'img_src'} + elems = self.parser.getElementsByTag(doc, **link_img_src_kwargs) + try_three = elems[0].get('href') if elems else None + + if try_three is None: + try_four = self.get_meta_content(doc, 'meta[name="og:image"]') + + top_meta_image = try_one or try_two or try_three or try_four # :) + + return urlparse.urljoin(article.url, top_meta_image) + + def get_meta_type(self, article): + """ + Returns meta type of article, open graph protocol. + """ + return self.get_meta_content(article.clean_doc, 'meta[property="og:type"]') + + def get_meta_description(self, article_or_source): """ If the article has meta description set in the source, use that. """ - return self.get_meta_content(article.doc, "meta[name=description]") + # Since objects use this particular method and sources don't + # have a 'clean_doc' we just use doc + try: # "easier to ask for forgiveness than permission" + doc = article_or_source.clean_doc + except: + doc = article_or_source.doc + return self.get_meta_content(doc, "meta[name=description]") def get_meta_keywords(self, article): """ If the article has meta keywords set in the source, use that. """ - return self.get_meta_content(article.doc, "meta[name=keywords]") + return self.get_meta_content(article.clean_doc, "meta[name=keywords]") + + def get_meta_data(self, article): + data = defaultdict(dict) + props = self.parser.css_select(article.clean_doc, 'meta') + + for prop in props: + key = prop.attrib.get('property') + if not key: + key = prop.attrib.get('name') + + if not key: + continue + + key = key.split(':') + + value = prop.attrib.get('content') + if not value: + value = prop.attrib.get('value') + + if not value: + continue + + value = value.strip() + + if value.isdigit(): + value = int(value) + + ref = data[key.pop(0)] + + for idx, part in enumerate(key): + if not key[idx:-1]: # no next values + ref[part] = value + break + if not ref.get(part): + ref[part] = dict() + else: + if isinstance(ref.get(part), basestring): + ref[part] = {'url': ref[part]} + ref = ref[part] + + return data def get_canonical_link(self, article): """ If the article has meta canonical link set in the url. """ kwargs = {'tag': 'link', 'attr': 'rel', 'value': 'canonical'} - meta = self.parser.getElementsByTag(article.doc, **kwargs) + meta = self.parser.getElementsByTag(article.clean_doc, **kwargs) if meta is not None and len(meta) > 0: href = self.parser.getAttribute(meta[0], 'href') if href: @@ -300,22 +414,74 @@ def get_canonical_link(self, article): return href return u'' - def get_img_urls(self, article): + def get_img_urls(self, article, use_top_node=False): """ Return all of the images on an html page, lxml root. """ - doc = article.raw_doc - urls = self.parser.get_img_urls(doc) - img_links = [ urlparse.urljoin(article.url, url) for url in urls ] + doc = article.clean_top_node if use_top_node else article.clean_doc + + img_kwargs = {'tag': 'img'} + img_tags = self.parser.getElementsByTag(doc, **img_kwargs) + urls = [img_tag.get('src') for img_tag in img_tags if img_tag.get('src')] + img_links = set([ urlparse.urljoin(article.url, url) for url in urls ]) + if article.meta_img: + img_links.add(article.meta_img) return img_links - def get_top_img_url(self, article): + def get_first_img_url(self, article): """ + Retrieves the first image in the 'top_node' + The top node is essentially the HTML + markdown where the main article lies and the first image + in that area is probably signifigcant. """ - # !important, we must use raw_doc because at this point doc has been cleaned - doc = article.raw_doc - return self.parser.get_top_img_url(doc) + node_images = self.get_img_urls(article, use_top_node=True) + node_images = list(node_images) + if node_images: + return urlparse.urljoin(article.url, node_images[0]) + return u'' + + def _get_urls(self, doc, titles): + """ + Return a list of urls or a list of (url, title_text) tuples if specified. + """ + if doc is None: + return [] + + a_kwargs = {'tag': 'a'} + a_tags = self.parser.getElementsByTag(doc, **a_kwargs) # doc.xpath('//a') + + # TODO this should be refactored! We should have a seperate method which + # siphones the titles our of a list of tags. + if titles: + return [ (a.get('href'), a.text) for a in a_tags if a.get('href') ] + + return [ a.get('href') for a in a_tags if a.get('href') ] + + def get_urls(self, doc_or_html, titles=False, regex=False): + """ + doc_or_htmls html page or doc and returns list of urls, the regex + flag indicates we don't parse via lxml and just search the html. + """ + if doc_or_html is None: + log.critical('Must extract urls from either html, text or doc!') + return [] + + # If we are extracting from raw text + if regex: + doc_or_html = re.sub('<[^<]+?>', ' ', doc_or_html) + doc_or_html = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', doc_or_html) + doc_or_html = [i.strip() for i in doc_or_html] + return doc_or_html or [] + + # If the doc_or_html is html, parse it into a root + if isinstance(doc_or_html, str) or isinstance(doc_or_html, unicode): + doc = self.parser.fromstring(doc_or_html) + else: + doc = doc_or_html + + return self._get_urls(doc, titles) def get_category_urls(self, source, source_url=None, page_urls=None): """ @@ -326,7 +492,7 @@ def get_category_urls(self, source, source_url=None, page_urls=None): """ source_url = source.url if not source_url else source_url - page_urls = self.parser.get_urls(source.doc) if not page_urls else page_urls + page_urls = self.get_urls(source.doc) if not page_urls else page_urls valid_categories = [] for p_url in page_urls: scheme = get_scheme(p_url, allow_fragments=False) @@ -439,24 +605,8 @@ def get_category_urls(self, source, source_url=None, page_urls=None): category_urls = [c for c in category_urls if c is not None] return category_urls - def get_feed_urls(self, source): - """ - Requires: List of category lxml roots, two types of anchors: categories - and feeds (rss). we extract category urls first and then feeds. - """ - feed_urls = [] - for category in source.categories: - root = category.doc - feed_urls.extend(self.parser.get_feed_urls(root)) - - feed_urls = feed_urls[:50] - feed_urls = [ prepare_url(f, source.url) for f in feed_urls ] - - feeds = list(set(feed_urls)) - return feeds - def extract_tags(self, article): - node = article.doc + node = article.clean_doc # node doesn't have chidren if len(list(node)) == 0: @@ -478,6 +628,11 @@ def extract_tags(self, article): def calculate_best_node(self, article): doc = article.doc + + # if article.config.hints: TODO: Maybe a future release + # rules_kw = article.config.hints.get(article.source_url) + # doc = article.parser.getElementsByTag(doc, **rules_kw)[0] + top_node = None nodes_to_check = self.nodes_to_check(doc) @@ -510,7 +665,7 @@ def calculate_best_node(self, article): if (nodes_number - i) <= bottom_negativescore_nodes: booster = float(bottom_negativescore_nodes - (nodes_number - i)) boost_score = float(-pow(booster, float(2))) - negscore = -abs(boost_score) + negative_scoring + negscore = abs(boost_score) + negative_scoring if negscore > 40: boost_score = float(5) diff --git a/newspaper/images.py b/newspaper/images.py index d2a095d5..9ceae28e 100644 --- a/newspaper/images.py +++ b/newspaper/images.py @@ -1,4 +1,12 @@ # -*- coding: utf-8 -*- +""" +The following image extraction implementation was taken from an old +copy of Reddit's source code. +""" +__title__ = 'newspaper' +__author__ = 'Lucas Ou-Yang' +__license__ = 'MIT' +__copyright__ = 'Copyright 2014, Lucas Ou-Yang' import logging import urllib @@ -9,10 +17,13 @@ from urllib2 import Request, HTTPError, URLError, build_opener from httplib import InvalidURL +from . import urls + log = logging.getLogger(__name__) chunk_size = 1024 thumbnail_size = 90, 90 +minimal_area = 5000 def image_to_str(image): s = StringIO.StringIO() @@ -113,7 +124,25 @@ def fetch_url(url, useragent, referer=None, retries=1, dimension=False): # pil failed to install, jpeg codec broken # **should work if you install via pillow print ('***jpeg misconfiguration! check pillow or pil' - 'installation this machine: %s' % str(e)) + 'installation this machine: %s' % str(e)) + p = None + break + except ValueError, ve: + log.debug('cant read image format: %s' % url) + p = None + break + except Exception, e: + # For some favicon.ico images, the image is so small + # that our PIL feed() method fails a length test. + # We add a check below for this. + is_favicon = (urls.url_to_filetype(url) == 'ico') + if is_favicon: + print 'we caught a favicon!: %s' % url + else: + # import traceback + # print traceback.format_exc() + print 'PIL feed() failure for image:', url, str(e) + raise e p = None break new_data = open_req.read(chunk_size) @@ -141,7 +170,7 @@ def fetch_url(url, useragent, referer=None, retries=1, dimension=False): if 'open_req' in locals(): open_req.close() -def fetch_size(url, useragent, referer=None, retries=1): +def fetch_image_dimension(url, useragent, referer=None, retries=1): return fetch_url(url, useragent, referer, retries, dimension=True) class Scraper: @@ -154,9 +183,9 @@ def __init__(self, article): self.useragent = self.config.browser_user_agent def largest_image_url(self): + #todo: remove. it is not responsibility of Scrapper if not self.imgs and not self.top_img: return None - if self.top_img: return self.top_img @@ -164,31 +193,8 @@ def largest_image_url(self): max_url = None for img_url in self.imgs: - size = fetch_size(img_url, self.useragent, referer=self.url) - if not size: - continue - - area = size[0] * size[1] - - # ignore little images - if area < 5000: - log.debug('ignore little %s' % img_url) - continue - - # PIL won't scale up, so we set a min width and - # maintain the aspect ratio - if size[0] < thumbnail_size[0]: - continue - - # ignore excessively long/wide images - if max(size) / min(size) > 1.5: - log.debug('ignore dims %s' % img_url) - continue - - # penalize images with "sprite" in their name - if 'sprite' in img_url.lower(): - log.debug('penalizing sprite %s' % img_url) - area /= 10 + dimension = fetch_image_dimension(img_url, self.useragent, referer=self.url) + area = self.calculate_area(img_url, dimension) if area > max_area: max_area = area @@ -197,6 +203,41 @@ def largest_image_url(self): log.debug('using max img ' + max_url) return max_url + def calculate_area(self, img_url, dimension): + if not dimension: + return 0 + + area = dimension[0] * dimension[1] + + #todo: introduce filter classes for each case + # ignore little images + if area < minimal_area: + log.debug('ignore little %s' % img_url) + return 0 + + # PIL won't scale up, so we set a min width and + # maintain the aspect ratio + if dimension[0] < thumbnail_size[0]: + return 0 + + # ignore excessively long/wide images + if max(dimension) / min(dimension) > self.config.image_dimension_ration: + log.debug('ignore dims %s' % img_url) + return 0 + + # penalize images with "sprite" in their name + lower_case_url = img_url.lower() + if 'sprite' in lower_case_url or 'logo' in lower_case_url: + log.debug('penalizing sprite %s' % img_url) + area /= 10 + + return area + + def satisfies_requirements(self, img_url): + dimension = fetch_image_dimension(img_url, self.useragent, referer=self.url) + area = self.calculate_area(img_url, dimension) + return area > minimal_area + def thumbnail(self): """ Identifies top image, trims out a thumbnail and also has a url. diff --git a/newspaper/mthreading.py b/newspaper/mthreading.py index 52960aab..b4837602 100644 --- a/newspaper/mthreading.py +++ b/newspaper/mthreading.py @@ -4,6 +4,11 @@ must be abstracted in this file. If we decide to do gevent also, it will deserve its own gevent file. """ +__title__ = 'newspaper' +__author__ = 'Lucas Ou-Yang' +__license__ = 'MIT' +__copyright__ = 'Copyright 2014, Lucas Ou-Yang' + import Queue from threading import Thread diff --git a/newspaper/network.py b/newspaper/network.py index 77e03051..94e1f214 100644 --- a/newspaper/network.py +++ b/newspaper/network.py @@ -1,13 +1,19 @@ # -*- coding: utf-8 -*- """ +All code involving requests and responses over the http network +must be abstracted in this file. """ +__title__ = 'newspaper' +__author__ = 'Lucas Ou-Yang' +__license__ = 'MIT' +__copyright__ = 'Copyright 2014, Lucas Ou-Yang' + import logging import requests from .settings import cj from .configuration import Configuration from .mthreading import ThreadPool -# from .packages import grequests log = logging.getLogger(__name__) diff --git a/newspaper/nlp.py b/newspaper/nlp.py index 6fc4b2df..0425e969 100644 --- a/newspaper/nlp.py +++ b/newspaper/nlp.py @@ -1,4 +1,11 @@ # -*- coding: utf-8 -*- +""" +Anything natural language related should be abstracted into this file. +""" +__title__ = 'newspaper' +__author__ = 'Lucas Ou-Yang' +__license__ = 'MIT' +__copyright__ = 'Copyright 2014, Lucas Ou-Yang' import re import math @@ -44,7 +51,7 @@ def score(sentences, titleWords, keywords): for i, s in enumerate(sentences): sentence = split_words(s) titleFeature = title_score(titleWords, sentence) - sentenceLength = length_score(sentence) + sentenceLength = length_score(len(sentence)) sentencePosition = sentence_position(i+1, senSize) sbsFeature = sbs(sentence, keywords) dbsFeature = dbs(sentence, keywords) @@ -140,10 +147,10 @@ def split_sentences(text): sentences = [x.replace('\n','') for x in sentences if len(x)>10] return sentences -def length_score(sentence): +def length_score(sentence_len): """ """ - return 1- math.fabs(ideal - len(sentence)) / ideal + return 1- math.fabs(ideal - sentence_len) / ideal def title_score(title, sentence): """ @@ -161,26 +168,28 @@ def sentence_position(i, size): probability of being an important sentence. """ normalized = i*1.0 / size - if (normalized > 0 and normalized <= 0.1): - return 0.17 - elif normalized > 0.1 and normalized <= 0.2: - return 0.23 - elif (normalized > 0.2 and normalized <= 0.3): - return 0.14 - elif (normalized > 0.3 and normalized <= 0.4): - return 0.08 - elif (normalized > 0.4 and normalized <= 0.5): - return 0.05 - elif (normalized > 0.5 and normalized <= 0.6): + if (normalized > 1.0): #just in case + return 0 + elif (normalized > 0.9): + return 0.15 + elif (normalized > 0.8): return 0.04 - elif (normalized > 0.6 and normalized <= 0.7): - return 0.06 - elif (normalized > 0.7 and normalized <= 0.8): + elif (normalized > 0.7): return 0.04 - elif (normalized > 0.8 and normalized <= 0.9): + elif (normalized > 0.6): + return 0.06 + elif (normalized > 0.5): return 0.04 - elif (normalized > 0.9 and normalized <= 1.0): - return 0.15 + elif (normalized > 0.4): + return 0.05 + elif (normalized > 0.3): + return 0.08 + elif (normalized > 0.2): + return 0.14 + elif (normalized > 0.1): + return 0.23 + elif (normalized > 0): + return 0.17 else: return 0 diff --git a/newspaper/outputformatters.py b/newspaper/outputformatters.py index 366526b0..267c815b 100644 --- a/newspaper/outputformatters.py +++ b/newspaper/outputformatters.py @@ -1,7 +1,12 @@ # -*- coding: utf-8 -*- - """ +Output formatting to text via lxml xpath nodes abstracted in this file. """ +__title__ = 'newspaper' +__author__ = 'Lucas Ou-Yang' +__license__ = 'MIT' +__copyright__ = 'Copyright 2014, Lucas Ou-Yang' + from HTMLParser import HTMLParser from .text import innerTrim diff --git a/newspaper/packages/BeautifulSoup.py b/newspaper/packages/BeautifulSoup.py deleted file mode 100644 index 4b17b853..00000000 --- a/newspaper/packages/BeautifulSoup.py +++ /dev/null @@ -1,2014 +0,0 @@ -"""Beautiful Soup -Elixir and Tonic -"The Screen-Scraper's Friend" -http://www.crummy.com/software/BeautifulSoup/ - -Beautiful Soup parses a (possibly invalid) XML or HTML document into a -tree representation. It provides methods and Pythonic idioms that make -it easy to navigate, search, and modify the tree. - -A well-formed XML/HTML document yields a well-formed data -structure. An ill-formed XML/HTML document yields a correspondingly -ill-formed data structure. If your document is only locally -well-formed, you can use this library to find and process the -well-formed part of it. - -Beautiful Soup works with Python 2.2 and up. It has no external -dependencies, but you'll have more success at converting data to UTF-8 -if you also install these three packages: - -* chardet, for auto-detecting character encodings - http://chardet.feedparser.org/ -* cjkcodecs and iconv_codec, which add more encodings to the ones supported - by stock Python. - http://cjkpython.i18n.org/ - -Beautiful Soup defines classes for two main parsing strategies: - - * BeautifulStoneSoup, for parsing XML, SGML, or your domain-specific - language that kind of looks like XML. - - * BeautifulSoup, for parsing run-of-the-mill HTML code, be it valid - or invalid. This class has web browser-like heuristics for - obtaining a sensible parse tree in the face of common HTML errors. - -Beautiful Soup also defines a class (UnicodeDammit) for autodetecting -the encoding of an HTML or XML document, and converting it to -Unicode. Much of this code is taken from Mark Pilgrim's Universal Feed Parser. - -For more than you ever wanted to know about Beautiful Soup, see the -documentation: -http://www.crummy.com/software/BeautifulSoup/documentation.html - -Here, have some legalese: - -Copyright (c) 2004-2010, Leonard Richardson - -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - * Redistributions in binary form must reproduce the above - copyright notice, this list of conditions and the following - disclaimer in the documentation and/or other materials provided - with the distribution. - - * Neither the name of the the Beautiful Soup Consortium and All - Night Kosher Bakery nor the names of its contributors may be - used to endorse or promote products derived from this software - without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR -CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, -EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, -PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR -PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF -LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING -NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE, DAMMIT. - -""" -from __future__ import generators - -__author__ = "Leonard Richardson (leonardr@segfault.org)" -__version__ = "3.2.0" -__copyright__ = "Copyright (c) 2004-2010 Leonard Richardson" -__license__ = "New-style BSD" - -from sgmllib import SGMLParser, SGMLParseError -import codecs -import markupbase -import types -import re -import sgmllib -try: - from htmlentitydefs import name2codepoint -except ImportError: - name2codepoint = {} -try: - set -except NameError: - from sets import Set as set - -#These hacks make Beautiful Soup able to parse XML with namespaces -sgmllib.tagfind = re.compile('[a-zA-Z][-_.:a-zA-Z0-9]*') -markupbase._declname_match = re.compile(r'[a-zA-Z][-_.:a-zA-Z0-9]*\s*').match - -DEFAULT_OUTPUT_ENCODING = "utf-8" - -def _match_css_class(str): - """Build a RE to match the given CSS class.""" - return re.compile(r"(^|.*\s)%s($|\s)" % str) - -# First, the classes that represent markup elements. - -class PageElement(object): - """Contains the navigational information for some part of the page - (either a tag or a piece of text)""" - - def setup(self, parent=None, previous=None): - """Sets up the initial relations between this element and - other elements.""" - self.parent = parent - self.previous = previous - self.next = None - self.previousSibling = None - self.nextSibling = None - if self.parent and self.parent.contents: - self.previousSibling = self.parent.contents[-1] - self.previousSibling.nextSibling = self - - def replaceWith(self, replaceWith): - oldParent = self.parent - myIndex = self.parent.index(self) - if hasattr(replaceWith, "parent")\ - and replaceWith.parent is self.parent: - # We're replacing this element with one of its siblings. - index = replaceWith.parent.index(replaceWith) - if index and index < myIndex: - # Furthermore, it comes before this element. That - # means that when we extract it, the index of this - # element will change. - myIndex = myIndex - 1 - self.extract() - oldParent.insert(myIndex, replaceWith) - - def replaceWithChildren(self): - myParent = self.parent - myIndex = self.parent.index(self) - self.extract() - reversedChildren = list(self.contents) - reversedChildren.reverse() - for child in reversedChildren: - myParent.insert(myIndex, child) - - def extract(self): - """Destructively rips this element out of the tree.""" - if self.parent: - try: - del self.parent.contents[self.parent.index(self)] - except ValueError: - pass - - #Find the two elements that would be next to each other if - #this element (and any children) hadn't been parsed. Connect - #the two. - lastChild = self._lastRecursiveChild() - nextElement = lastChild.next - - if self.previous: - self.previous.next = nextElement - if nextElement: - nextElement.previous = self.previous - self.previous = None - lastChild.next = None - - self.parent = None - if self.previousSibling: - self.previousSibling.nextSibling = self.nextSibling - if self.nextSibling: - self.nextSibling.previousSibling = self.previousSibling - self.previousSibling = self.nextSibling = None - return self - - def _lastRecursiveChild(self): - "Finds the last element beneath this object to be parsed." - lastChild = self - while hasattr(lastChild, 'contents') and lastChild.contents: - lastChild = lastChild.contents[-1] - return lastChild - - def insert(self, position, newChild): - if isinstance(newChild, basestring) \ - and not isinstance(newChild, NavigableString): - newChild = NavigableString(newChild) - - position = min(position, len(self.contents)) - if hasattr(newChild, 'parent') and newChild.parent is not None: - # We're 'inserting' an element that's already one - # of this object's children. - if newChild.parent is self: - index = self.index(newChild) - if index > position: - # Furthermore we're moving it further down the - # list of this object's children. That means that - # when we extract this element, our target index - # will jump down one. - position = position - 1 - newChild.extract() - - newChild.parent = self - previousChild = None - if position == 0: - newChild.previousSibling = None - newChild.previous = self - else: - previousChild = self.contents[position-1] - newChild.previousSibling = previousChild - newChild.previousSibling.nextSibling = newChild - newChild.previous = previousChild._lastRecursiveChild() - if newChild.previous: - newChild.previous.next = newChild - - newChildsLastElement = newChild._lastRecursiveChild() - - if position >= len(self.contents): - newChild.nextSibling = None - - parent = self - parentsNextSibling = None - while not parentsNextSibling: - parentsNextSibling = parent.nextSibling - parent = parent.parent - if not parent: # This is the last element in the document. - break - if parentsNextSibling: - newChildsLastElement.next = parentsNextSibling - else: - newChildsLastElement.next = None - else: - nextChild = self.contents[position] - newChild.nextSibling = nextChild - if newChild.nextSibling: - newChild.nextSibling.previousSibling = newChild - newChildsLastElement.next = nextChild - - if newChildsLastElement.next: - newChildsLastElement.next.previous = newChildsLastElement - self.contents.insert(position, newChild) - - def append(self, tag): - """Appends the given tag to the contents of this tag.""" - self.insert(len(self.contents), tag) - - def findNext(self, name=None, attrs={}, text=None, **kwargs): - """Returns the first item that matches the given criteria and - appears after this Tag in the document.""" - return self._findOne(self.findAllNext, name, attrs, text, **kwargs) - - def findAllNext(self, name=None, attrs={}, text=None, limit=None, - **kwargs): - """Returns all items that match the given criteria and appear - after this Tag in the document.""" - return self._findAll(name, attrs, text, limit, self.nextGenerator, - **kwargs) - - def findNextSibling(self, name=None, attrs={}, text=None, **kwargs): - """Returns the closest sibling to this Tag that matches the - given criteria and appears after this Tag in the document.""" - return self._findOne(self.findNextSiblings, name, attrs, text, - **kwargs) - - def findNextSiblings(self, name=None, attrs={}, text=None, limit=None, - **kwargs): - """Returns the siblings of this Tag that match the given - criteria and appear after this Tag in the document.""" - return self._findAll(name, attrs, text, limit, - self.nextSiblingGenerator, **kwargs) - fetchNextSiblings = findNextSiblings # Compatibility with pre-3.x - - def findPrevious(self, name=None, attrs={}, text=None, **kwargs): - """Returns the first item that matches the given criteria and - appears before this Tag in the document.""" - return self._findOne(self.findAllPrevious, name, attrs, text, **kwargs) - - def findAllPrevious(self, name=None, attrs={}, text=None, limit=None, - **kwargs): - """Returns all items that match the given criteria and appear - before this Tag in the document.""" - return self._findAll(name, attrs, text, limit, self.previousGenerator, - **kwargs) - fetchPrevious = findAllPrevious # Compatibility with pre-3.x - - def findPreviousSibling(self, name=None, attrs={}, text=None, **kwargs): - """Returns the closest sibling to this Tag that matches the - given criteria and appears before this Tag in the document.""" - return self._findOne(self.findPreviousSiblings, name, attrs, text, - **kwargs) - - def findPreviousSiblings(self, name=None, attrs={}, text=None, - limit=None, **kwargs): - """Returns the siblings of this Tag that match the given - criteria and appear before this Tag in the document.""" - return self._findAll(name, attrs, text, limit, - self.previousSiblingGenerator, **kwargs) - fetchPreviousSiblings = findPreviousSiblings # Compatibility with pre-3.x - - def findParent(self, name=None, attrs={}, **kwargs): - """Returns the closest parent of this Tag that matches the given - criteria.""" - # NOTE: We can't use _findOne because findParents takes a different - # set of arguments. - r = None - l = self.findParents(name, attrs, 1) - if l: - r = l[0] - return r - - def findParents(self, name=None, attrs={}, limit=None, **kwargs): - """Returns the parents of this Tag that match the given - criteria.""" - - return self._findAll(name, attrs, None, limit, self.parentGenerator, - **kwargs) - fetchParents = findParents # Compatibility with pre-3.x - - #These methods do the real heavy lifting. - - def _findOne(self, method, name, attrs, text, **kwargs): - r = None - l = method(name, attrs, text, 1, **kwargs) - if l: - r = l[0] - return r - - def _findAll(self, name, attrs, text, limit, generator, **kwargs): - "Iterates over a generator looking for things that match." - - if isinstance(name, SoupStrainer): - strainer = name - # (Possibly) special case some findAll*(...) searches - elif text is None and not limit and not attrs and not kwargs: - # findAll*(True) - if name is True: - return [element for element in generator() - if isinstance(element, Tag)] - # findAll*('tag-name') - elif isinstance(name, basestring): - return [element for element in generator() - if isinstance(element, Tag) and - element.name == name] - else: - strainer = SoupStrainer(name, attrs, text, **kwargs) - # Build a SoupStrainer - else: - strainer = SoupStrainer(name, attrs, text, **kwargs) - results = ResultSet(strainer) - g = generator() - while True: - try: - i = g.next() - except StopIteration: - break - if i: - found = strainer.search(i) - if found: - results.append(found) - if limit and len(results) >= limit: - break - return results - - #These Generators can be used to navigate starting from both - #NavigableStrings and Tags. - def nextGenerator(self): - i = self - while i is not None: - i = i.next - yield i - - def nextSiblingGenerator(self): - i = self - while i is not None: - i = i.nextSibling - yield i - - def previousGenerator(self): - i = self - while i is not None: - i = i.previous - yield i - - def previousSiblingGenerator(self): - i = self - while i is not None: - i = i.previousSibling - yield i - - def parentGenerator(self): - i = self - while i is not None: - i = i.parent - yield i - - # Utility methods - def substituteEncoding(self, str, encoding=None): - encoding = encoding or "utf-8" - return str.replace("%SOUP-ENCODING%", encoding) - - def toEncoding(self, s, encoding=None): - """Encodes an object to a string in some encoding, or to Unicode. - .""" - if isinstance(s, unicode): - if encoding: - s = s.encode(encoding) - elif isinstance(s, str): - if encoding: - s = s.encode(encoding) - else: - s = unicode(s) - else: - if encoding: - s = self.toEncoding(str(s), encoding) - else: - s = unicode(s) - return s - -class NavigableString(unicode, PageElement): - - def __new__(cls, value): - """Create a new NavigableString. - - When unpickling a NavigableString, this method is called with - the string in DEFAULT_OUTPUT_ENCODING. That encoding needs to be - passed in to the superclass's __new__ or the superclass won't know - how to handle non-ASCII characters. - """ - if isinstance(value, unicode): - return unicode.__new__(cls, value) - return unicode.__new__(cls, value, DEFAULT_OUTPUT_ENCODING) - - def __getnewargs__(self): - return (NavigableString.__str__(self),) - - def __getattr__(self, attr): - """text.string gives you text. This is for backwards - compatibility for Navigable*String, but for CData* it lets you - get the string without the CData wrapper.""" - if attr == 'string': - return self - else: - raise AttributeError, "'%s' object has no attribute '%s'" % (self.__class__.__name__, attr) - - def __unicode__(self): - return str(self).decode(DEFAULT_OUTPUT_ENCODING) - - def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING): - if encoding: - return self.encode(encoding) - else: - return self - -class CData(NavigableString): - - def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING): - return "" % NavigableString.__str__(self, encoding) - -class ProcessingInstruction(NavigableString): - def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING): - output = self - if "%SOUP-ENCODING%" in output: - output = self.substituteEncoding(output, encoding) - return "" % self.toEncoding(output, encoding) - -class Comment(NavigableString): - def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING): - return "" % NavigableString.__str__(self, encoding) - -class Declaration(NavigableString): - def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING): - return "" % NavigableString.__str__(self, encoding) - -class Tag(PageElement): - - """Represents a found HTML tag with its attributes and contents.""" - - def _invert(h): - "Cheap function to invert a hash." - i = {} - for k,v in h.items(): - i[v] = k - return i - - XML_ENTITIES_TO_SPECIAL_CHARS = { "apos" : "'", - "quot" : '"', - "amp" : "&", - "lt" : "<", - "gt" : ">" } - - XML_SPECIAL_CHARS_TO_ENTITIES = _invert(XML_ENTITIES_TO_SPECIAL_CHARS) - - def _convertEntities(self, match): - """Used in a call to re.sub to replace HTML, XML, and numeric - entities with the appropriate Unicode characters. If HTML - entities are being converted, any unrecognized entities are - escaped.""" - x = match.group(1) - if self.convertHTMLEntities and x in name2codepoint: - return unichr(name2codepoint[x]) - elif x in self.XML_ENTITIES_TO_SPECIAL_CHARS: - if self.convertXMLEntities: - return self.XML_ENTITIES_TO_SPECIAL_CHARS[x] - else: - return u'&%s;' % x - elif len(x) > 0 and x[0] == '#': - # Handle numeric entities - if len(x) > 1 and x[1] == 'x': - return unichr(int(x[2:], 16)) - else: - return unichr(int(x[1:])) - - elif self.escapeUnrecognizedEntities: - return u'&%s;' % x - else: - return u'&%s;' % x - - def __init__(self, parser, name, attrs=None, parent=None, - previous=None): - "Basic constructor." - - # We don't actually store the parser object: that lets extracted - # chunks be garbage-collected - self.parserClass = parser.__class__ - self.isSelfClosing = parser.isSelfClosingTag(name) - self.name = name - if attrs is None: - attrs = [] - elif isinstance(attrs, dict): - attrs = attrs.items() - self.attrs = attrs - self.contents = [] - self.setup(parent, previous) - self.hidden = False - self.containsSubstitutions = False - self.convertHTMLEntities = parser.convertHTMLEntities - self.convertXMLEntities = parser.convertXMLEntities - self.escapeUnrecognizedEntities = parser.escapeUnrecognizedEntities - - # Convert any HTML, XML, or numeric entities in the attribute values. - convert = lambda(k, val): (k, - re.sub("&(#\d+|#x[0-9a-fA-F]+|\w+);", - self._convertEntities, - val)) - self.attrs = map(convert, self.attrs) - - def getString(self): - if (len(self.contents) == 1 - and isinstance(self.contents[0], NavigableString)): - return self.contents[0] - - def setString(self, string): - """Replace the contents of the tag with a string""" - self.clear() - self.append(string) - - string = property(getString, setString) - - def getText(self, separator=u""): - if not len(self.contents): - return u"" - stopNode = self._lastRecursiveChild().next - strings = [] - current = self.contents[0] - while current is not stopNode: - if isinstance(current, NavigableString): - strings.append(current.strip()) - current = current.next - return separator.join(strings) - - text = property(getText) - - def get(self, key, default=None): - """Returns the value of the 'key' attribute for the tag, or - the value given for 'default' if it doesn't have that - attribute.""" - return self._getAttrMap().get(key, default) - - def clear(self): - """Extract all children.""" - for child in self.contents[:]: - child.extract() - - def index(self, element): - for i, child in enumerate(self.contents): - if child is element: - return i - raise ValueError("Tag.index: element not in tag") - - def has_key(self, key): - return self._getAttrMap().has_key(key) - - def __getitem__(self, key): - """tag[key] returns the value of the 'key' attribute for the tag, - and throws an exception if it's not there.""" - return self._getAttrMap()[key] - - def __iter__(self): - "Iterating over a tag iterates over its contents." - return iter(self.contents) - - def __len__(self): - "The length of a tag is the length of its list of contents." - return len(self.contents) - - def __contains__(self, x): - return x in self.contents - - def __nonzero__(self): - "A tag is non-None even if it has no contents." - return True - - def __setitem__(self, key, value): - """Setting tag[key] sets the value of the 'key' attribute for the - tag.""" - self._getAttrMap() - self.attrMap[key] = value - found = False - for i in range(0, len(self.attrs)): - if self.attrs[i][0] == key: - self.attrs[i] = (key, value) - found = True - if not found: - self.attrs.append((key, value)) - self._getAttrMap()[key] = value - - def __delitem__(self, key): - "Deleting tag[key] deletes all 'key' attributes for the tag." - for item in self.attrs: - if item[0] == key: - self.attrs.remove(item) - #We don't break because bad HTML can define the same - #attribute multiple times. - self._getAttrMap() - if self.attrMap.has_key(key): - del self.attrMap[key] - - def __call__(self, *args, **kwargs): - """Calling a tag like a function is the same as calling its - findAll() method. Eg. tag('a') returns a list of all the A tags - found within this tag.""" - return apply(self.findAll, args, kwargs) - - def __getattr__(self, tag): - #print "Getattr %s.%s" % (self.__class__, tag) - if len(tag) > 3 and tag.rfind('Tag') == len(tag)-3: - return self.find(tag[:-3]) - elif tag.find('__') != 0: - return self.find(tag) - raise AttributeError, "'%s' object has no attribute '%s'" % (self.__class__, tag) - - def __eq__(self, other): - """Returns true iff this tag has the same name, the same attributes, - and the same contents (recursively) as the given tag. - - NOTE: right now this will return false if two tags have the - same attributes in a different order. Should this be fixed?""" - if other is self: - return True - if not hasattr(other, 'name') or not hasattr(other, 'attrs') or not hasattr(other, 'contents') or self.name != other.name or self.attrs != other.attrs or len(self) != len(other): - return False - for i in range(0, len(self.contents)): - if self.contents[i] != other.contents[i]: - return False - return True - - def __ne__(self, other): - """Returns true iff this tag is not identical to the other tag, - as defined in __eq__.""" - return not self == other - - def __repr__(self, encoding=DEFAULT_OUTPUT_ENCODING): - """Renders this tag as a string.""" - return self.__str__(encoding) - - def __unicode__(self): - return self.__str__(None) - - BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|" - + "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)" - + ")") - - def _sub_entity(self, x): - """Used with a regular expression to substitute the - appropriate XML entity for an XML special character.""" - return "&" + self.XML_SPECIAL_CHARS_TO_ENTITIES[x.group(0)[0]] + ";" - - def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING, - prettyPrint=False, indentLevel=0): - """Returns a string or Unicode representation of this tag and - its contents. To get Unicode, pass None for encoding. - - NOTE: since Python's HTML parser consumes whitespace, this - method is not certain to reproduce the whitespace present in - the original string.""" - - encodedName = self.toEncoding(self.name, encoding) - - attrs = [] - if self.attrs: - for key, val in self.attrs: - fmt = '%s="%s"' - if isinstance(val, basestring): - if self.containsSubstitutions and '%SOUP-ENCODING%' in val: - val = self.substituteEncoding(val, encoding) - - # The attribute value either: - # - # * Contains no embedded double quotes or single quotes. - # No problem: we enclose it in double quotes. - # * Contains embedded single quotes. No problem: - # double quotes work here too. - # * Contains embedded double quotes. No problem: - # we enclose it in single quotes. - # * Embeds both single _and_ double quotes. This - # can't happen naturally, but it can happen if - # you modify an attribute value after parsing - # the document. Now we have a bit of a - # problem. We solve it by enclosing the - # attribute in single quotes, and escaping any - # embedded single quotes to XML entities. - if '"' in val: - fmt = "%s='%s'" - if "'" in val: - # TODO: replace with apos when - # appropriate. - val = val.replace("'", "&squot;") - - # Now we're okay w/r/t quotes. But the attribute - # value might also contain angle brackets, or - # ampersands that aren't part of entities. We need - # to escape those to XML entities too. - val = self.BARE_AMPERSAND_OR_BRACKET.sub(self._sub_entity, val) - - attrs.append(fmt % (self.toEncoding(key, encoding), - self.toEncoding(val, encoding))) - close = '' - closeTag = '' - if self.isSelfClosing: - close = ' /' - else: - closeTag = '' % encodedName - - indentTag, indentContents = 0, 0 - if prettyPrint: - indentTag = indentLevel - space = (' ' * (indentTag-1)) - indentContents = indentTag + 1 - contents = self.renderContents(encoding, prettyPrint, indentContents) - if self.hidden: - s = contents - else: - s = [] - attributeString = '' - if attrs: - attributeString = ' ' + ' '.join(attrs) - if prettyPrint: - s.append(space) - s.append('<%s%s%s>' % (encodedName, attributeString, close)) - if prettyPrint: - s.append("\n") - s.append(contents) - if prettyPrint and contents and contents[-1] != "\n": - s.append("\n") - if prettyPrint and closeTag: - s.append(space) - s.append(closeTag) - if prettyPrint and closeTag and self.nextSibling: - s.append("\n") - s = ''.join(s) - return s - - def decompose(self): - """Recursively destroys the contents of this tree.""" - self.extract() - if len(self.contents) == 0: - return - current = self.contents[0] - while current is not None: - next = current.next - if isinstance(current, Tag): - del current.contents[:] - current.parent = None - current.previous = None - current.previousSibling = None - current.next = None - current.nextSibling = None - current = next - - def prettify(self, encoding=DEFAULT_OUTPUT_ENCODING): - return self.__str__(encoding, True) - - def renderContents(self, encoding=DEFAULT_OUTPUT_ENCODING, - prettyPrint=False, indentLevel=0): - """Renders the contents of this tag as a string in the given - encoding. If encoding is None, returns a Unicode string..""" - s=[] - for c in self: - text = None - if isinstance(c, NavigableString): - text = c.__str__(encoding) - elif isinstance(c, Tag): - s.append(c.__str__(encoding, prettyPrint, indentLevel)) - if text and prettyPrint: - text = text.strip() - if text: - if prettyPrint: - s.append(" " * (indentLevel-1)) - s.append(text) - if prettyPrint: - s.append("\n") - return ''.join(s) - - #Soup methods - - def find(self, name=None, attrs={}, recursive=True, text=None, - **kwargs): - """Return only the first child of this Tag matching the given - criteria.""" - r = None - l = self.findAll(name, attrs, recursive, text, 1, **kwargs) - if l: - r = l[0] - return r - findChild = find - - def findAll(self, name=None, attrs={}, recursive=True, text=None, - limit=None, **kwargs): - """Extracts a list of Tag objects that match the given - criteria. You can specify the name of the Tag and any - attributes you want the Tag to have. - - The value of a key-value pair in the 'attrs' map can be a - string, a list of strings, a regular expression object, or a - callable that takes a string and returns whether or not the - string matches for some custom definition of 'matches'. The - same is true of the tag name.""" - generator = self.recursiveChildGenerator - if not recursive: - generator = self.childGenerator - return self._findAll(name, attrs, text, limit, generator, **kwargs) - findChildren = findAll - - # Pre-3.x compatibility methods - first = find - fetch = findAll - - def fetchText(self, text=None, recursive=True, limit=None): - return self.findAll(text=text, recursive=recursive, limit=limit) - - def firstText(self, text=None, recursive=True): - return self.find(text=text, recursive=recursive) - - #Private methods - - def _getAttrMap(self): - """Initializes a map representation of this tag's attributes, - if not already initialized.""" - if not getattr(self, 'attrMap'): - self.attrMap = {} - for (key, value) in self.attrs: - self.attrMap[key] = value - return self.attrMap - - #Generator methods - def childGenerator(self): - # Just use the iterator from the contents - return iter(self.contents) - - def recursiveChildGenerator(self): - if not len(self.contents): - raise StopIteration - stopNode = self._lastRecursiveChild().next - current = self.contents[0] - while current is not stopNode: - yield current - current = current.next - - -# Next, a couple classes to represent queries and their results. -class SoupStrainer: - """Encapsulates a number of ways of matching a markup element (tag or - text).""" - - def __init__(self, name=None, attrs={}, text=None, **kwargs): - self.name = name - if isinstance(attrs, basestring): - kwargs['class'] = _match_css_class(attrs) - attrs = None - if kwargs: - if attrs: - attrs = attrs.copy() - attrs.update(kwargs) - else: - attrs = kwargs - self.attrs = attrs - self.text = text - - def __str__(self): - if self.text: - return self.text - else: - return "%s|%s" % (self.name, self.attrs) - - def searchTag(self, markupName=None, markupAttrs={}): - found = None - markup = None - if isinstance(markupName, Tag): - markup = markupName - markupAttrs = markup - callFunctionWithTagData = callable(self.name) \ - and not isinstance(markupName, Tag) - - if (not self.name) \ - or callFunctionWithTagData \ - or (markup and self._matches(markup, self.name)) \ - or (not markup and self._matches(markupName, self.name)): - if callFunctionWithTagData: - match = self.name(markupName, markupAttrs) - else: - match = True - markupAttrMap = None - for attr, matchAgainst in self.attrs.items(): - if not markupAttrMap: - if hasattr(markupAttrs, 'get'): - markupAttrMap = markupAttrs - else: - markupAttrMap = {} - for k,v in markupAttrs: - markupAttrMap[k] = v - attrValue = markupAttrMap.get(attr) - if not self._matches(attrValue, matchAgainst): - match = False - break - if match: - if markup: - found = markup - else: - found = markupName - return found - - def search(self, markup): - #print 'looking for %s in %s' % (self, markup) - found = None - # If given a list of items, scan it for a text element that - # matches. - if hasattr(markup, "__iter__") \ - and not isinstance(markup, Tag): - for element in markup: - if isinstance(element, NavigableString) \ - and self.search(element): - found = element - break - # If it's a Tag, make sure its name or attributes match. - # Don't bother with Tags if we're searching for text. - elif isinstance(markup, Tag): - if not self.text: - found = self.searchTag(markup) - # If it's text, make sure the text matches. - elif isinstance(markup, NavigableString) or \ - isinstance(markup, basestring): - if self._matches(markup, self.text): - found = markup - else: - raise Exception, "I don't know how to match against a %s" \ - % markup.__class__ - return found - - def _matches(self, markup, matchAgainst): - #print "Matching %s against %s" % (markup, matchAgainst) - result = False - if matchAgainst is True: - result = markup is not None - elif callable(matchAgainst): - result = matchAgainst(markup) - else: - #Custom match methods take the tag as an argument, but all - #other ways of matching match the tag name as a string. - if isinstance(markup, Tag): - markup = markup.name - if markup and not isinstance(markup, basestring): - markup = unicode(markup) - #Now we know that chunk is either a string, or None. - if hasattr(matchAgainst, 'match'): - # It's a regexp object. - result = markup and matchAgainst.search(markup) - elif hasattr(matchAgainst, '__iter__'): # list-like - result = markup in matchAgainst - elif hasattr(matchAgainst, 'items'): - result = markup.has_key(matchAgainst) - elif matchAgainst and isinstance(markup, basestring): - if isinstance(markup, unicode): - matchAgainst = unicode(matchAgainst) - else: - matchAgainst = str(matchAgainst) - - if not result: - result = matchAgainst == markup - return result - -class ResultSet(list): - """A ResultSet is just a list that keeps track of the SoupStrainer - that created it.""" - def __init__(self, source): - list.__init__([]) - self.source = source - -# Now, some helper functions. - -def buildTagMap(default, *args): - """Turns a list of maps, lists, or scalars into a single map. - Used to build the SELF_CLOSING_TAGS, NESTABLE_TAGS, and - NESTING_RESET_TAGS maps out of lists and partial maps.""" - built = {} - for portion in args: - if hasattr(portion, 'items'): - #It's a map. Merge it. - for k,v in portion.items(): - built[k] = v - elif hasattr(portion, '__iter__'): # is a list - #It's a list. Map each item to the default. - for k in portion: - built[k] = default - else: - #It's a scalar. Map it to the default. - built[portion] = default - return built - -# Now, the parser classes. - -class BeautifulStoneSoup(Tag, SGMLParser): - - """This class contains the basic parser and search code. It defines - a parser that knows nothing about tag behavior except for the - following: - - You can't close a tag without closing all the tags it encloses. - That is, "" actually means - "". - - [Another possible explanation is "", but since - this class defines no SELF_CLOSING_TAGS, it will never use that - explanation.] - - This class is useful for parsing XML or made-up markup languages, - or when BeautifulSoup makes an assumption counter to what you were - expecting.""" - - SELF_CLOSING_TAGS = {} - NESTABLE_TAGS = {} - RESET_NESTING_TAGS = {} - QUOTE_TAGS = {} - PRESERVE_WHITESPACE_TAGS = [] - - MARKUP_MASSAGE = [(re.compile('(<[^<>]*)/>'), - lambda x: x.group(1) + ' />'), - (re.compile(']*)>'), - lambda x: '') - ] - - ROOT_TAG_NAME = u'[document]' - - HTML_ENTITIES = "html" - XML_ENTITIES = "xml" - XHTML_ENTITIES = "xhtml" - # TODO: This only exists for backwards-compatibility - ALL_ENTITIES = XHTML_ENTITIES - - # Used when determining whether a text node is all whitespace and - # can be replaced with a single space. A text node that contains - # fancy Unicode spaces (usually non-breaking) should be left - # alone. - STRIP_ASCII_SPACES = { 9: None, 10: None, 12: None, 13: None, 32: None, } - - def __init__(self, markup="", parseOnlyThese=None, fromEncoding=None, - markupMassage=True, smartQuotesTo=XML_ENTITIES, - convertEntities=None, selfClosingTags=None, isHTML=False): - """The Soup object is initialized as the 'root tag', and the - provided markup (which can be a string or a file-like object) - is fed into the underlying parser. - - sgmllib will process most bad HTML, and the BeautifulSoup - class has some tricks for dealing with some HTML that kills - sgmllib, but Beautiful Soup can nonetheless choke or lose data - if your data uses self-closing tags or declarations - incorrectly. - - By default, Beautiful Soup uses regexes to sanitize input, - avoiding the vast majority of these problems. If the problems - don't apply to you, pass in False for markupMassage, and - you'll get better performance. - - The default parser massage techniques fix the two most common - instances of invalid HTML that choke sgmllib: - -
(No space between name of closing tag and tag close) - (Extraneous whitespace in declaration) - - You can pass in a custom list of (RE object, replace method) - tuples to get Beautiful Soup to scrub your input the way you - want.""" - - self.parseOnlyThese = parseOnlyThese - self.fromEncoding = fromEncoding - self.smartQuotesTo = smartQuotesTo - self.convertEntities = convertEntities - # Set the rules for how we'll deal with the entities we - # encounter - if self.convertEntities: - # It doesn't make sense to convert encoded characters to - # entities even while you're converting entities to Unicode. - # Just convert it all to Unicode. - self.smartQuotesTo = None - if convertEntities == self.HTML_ENTITIES: - self.convertXMLEntities = False - self.convertHTMLEntities = True - self.escapeUnrecognizedEntities = True - elif convertEntities == self.XHTML_ENTITIES: - self.convertXMLEntities = True - self.convertHTMLEntities = True - self.escapeUnrecognizedEntities = False - elif convertEntities == self.XML_ENTITIES: - self.convertXMLEntities = True - self.convertHTMLEntities = False - self.escapeUnrecognizedEntities = False - else: - self.convertXMLEntities = False - self.convertHTMLEntities = False - self.escapeUnrecognizedEntities = False - - self.instanceSelfClosingTags = buildTagMap(None, selfClosingTags) - SGMLParser.__init__(self) - - if hasattr(markup, 'read'): # It's a file-type object. - markup = markup.read() - self.markup = markup - self.markupMassage = markupMassage - try: - self._feed(isHTML=isHTML) - except StopParsing: - pass - self.markup = None # The markup can now be GCed - - def convert_charref(self, name): - """This method fixes a bug in Python's SGMLParser.""" - try: - n = int(name) - except ValueError: - return - if not 0 <= n <= 127 : # ASCII ends at 127, not 255 - return - return self.convert_codepoint(n) - - def _feed(self, inDocumentEncoding=None, isHTML=False): - # Convert the document to Unicode. - markup = self.markup - if isinstance(markup, unicode): - if not hasattr(self, 'originalEncoding'): - self.originalEncoding = None - else: - dammit = UnicodeDammit\ - (markup, [self.fromEncoding, inDocumentEncoding], - smartQuotesTo=self.smartQuotesTo, isHTML=isHTML) - markup = dammit.unicode - self.originalEncoding = dammit.originalEncoding - self.declaredHTMLEncoding = dammit.declaredHTMLEncoding - if markup: - if self.markupMassage: - if not hasattr(self.markupMassage, "__iter__"): - self.markupMassage = self.MARKUP_MASSAGE - for fix, m in self.markupMassage: - markup = fix.sub(m, markup) - # TODO: We get rid of markupMassage so that the - # soup object can be deepcopied later on. Some - # Python installations can't copy regexes. If anyone - # was relying on the existence of markupMassage, this - # might cause problems. - del(self.markupMassage) - self.reset() - - SGMLParser.feed(self, markup) - # Close out any unfinished strings and close all the open tags. - self.endData() - while self.currentTag.name != self.ROOT_TAG_NAME: - self.popTag() - - def __getattr__(self, methodName): - """This method routes method call requests to either the SGMLParser - superclass or the Tag superclass, depending on the method name.""" - #print "__getattr__ called on %s.%s" % (self.__class__, methodName) - - if methodName.startswith('start_') or methodName.startswith('end_') \ - or methodName.startswith('do_'): - return SGMLParser.__getattr__(self, methodName) - elif not methodName.startswith('__'): - return Tag.__getattr__(self, methodName) - else: - raise AttributeError - - def isSelfClosingTag(self, name): - """Returns true iff the given string is the name of a - self-closing tag according to this parser.""" - return self.SELF_CLOSING_TAGS.has_key(name) \ - or self.instanceSelfClosingTags.has_key(name) - - def reset(self): - Tag.__init__(self, self, self.ROOT_TAG_NAME) - self.hidden = 1 - SGMLParser.reset(self) - self.currentData = [] - self.currentTag = None - self.tagStack = [] - self.quoteStack = [] - self.pushTag(self) - - def popTag(self): - tag = self.tagStack.pop() - - #print "Pop", tag.name - if self.tagStack: - self.currentTag = self.tagStack[-1] - return self.currentTag - - def pushTag(self, tag): - #print "Push", tag.name - if self.currentTag: - self.currentTag.contents.append(tag) - self.tagStack.append(tag) - self.currentTag = self.tagStack[-1] - - def endData(self, containerClass=NavigableString): - if self.currentData: - currentData = u''.join(self.currentData) - if (currentData.translate(self.STRIP_ASCII_SPACES) == '' and - not set([tag.name for tag in self.tagStack]).intersection( - self.PRESERVE_WHITESPACE_TAGS)): - if '\n' in currentData: - currentData = '\n' - else: - currentData = ' ' - self.currentData = [] - if self.parseOnlyThese and len(self.tagStack) <= 1 and \ - (not self.parseOnlyThese.text or \ - not self.parseOnlyThese.search(currentData)): - return - o = containerClass(currentData) - o.setup(self.currentTag, self.previous) - if self.previous: - self.previous.next = o - self.previous = o - self.currentTag.contents.append(o) - - - def _popToTag(self, name, inclusivePop=True): - """Pops the tag stack up to and including the most recent - instance of the given tag. If inclusivePop is false, pops the tag - stack up to but *not* including the most recent instqance of - the given tag.""" - #print "Popping to %s" % name - if name == self.ROOT_TAG_NAME: - return - - numPops = 0 - mostRecentTag = None - for i in range(len(self.tagStack)-1, 0, -1): - if name == self.tagStack[i].name: - numPops = len(self.tagStack)-i - break - if not inclusivePop: - numPops = numPops - 1 - - for i in range(0, numPops): - mostRecentTag = self.popTag() - return mostRecentTag - - def _smartPop(self, name): - - """We need to pop up to the previous tag of this type, unless - one of this tag's nesting reset triggers comes between this - tag and the previous tag of this type, OR unless this tag is a - generic nesting trigger and another generic nesting trigger - comes between this tag and the previous tag of this type. - - Examples: -

FooBar *

* should pop to 'p', not 'b'. -

FooBar *

* should pop to 'table', not 'p'. -

Foo

Bar *

* should pop to 'tr', not 'p'. - -

    • *
    • * should pop to 'ul', not the first 'li'. -
    • ** should pop to 'table', not the first 'tr' - tag should - implicitly close the previous tag within the same
    ** should pop to 'tr', not the first 'td' - """ - - nestingResetTriggers = self.NESTABLE_TAGS.get(name) - isNestable = nestingResetTriggers != None - isResetNesting = self.RESET_NESTING_TAGS.has_key(name) - popTo = None - inclusive = True - for i in range(len(self.tagStack)-1, 0, -1): - p = self.tagStack[i] - if (not p or p.name == name) and not isNestable: - #Non-nestable tags get popped to the top or to their - #last occurance. - popTo = name - break - if (nestingResetTriggers is not None - and p.name in nestingResetTriggers) \ - or (nestingResetTriggers is None and isResetNesting - and self.RESET_NESTING_TAGS.has_key(p.name)): - - #If we encounter one of the nesting reset triggers - #peculiar to this tag, or we encounter another tag - #that causes nesting to reset, pop up to but not - #including that tag. - popTo = p.name - inclusive = False - break - p = p.parent - if popTo: - self._popToTag(popTo, inclusive) - - def unknown_starttag(self, name, attrs, selfClosing=0): - #print "Start tag %s: %s" % (name, attrs) - if self.quoteStack: - #This is not a real tag. - #print "<%s> is not real!" % name - attrs = ''.join([' %s="%s"' % (x, y) for x, y in attrs]) - self.handle_data('<%s%s>' % (name, attrs)) - return - self.endData() - - if not self.isSelfClosingTag(name) and not selfClosing: - self._smartPop(name) - - if self.parseOnlyThese and len(self.tagStack) <= 1 \ - and (self.parseOnlyThese.text or not self.parseOnlyThese.searchTag(name, attrs)): - return - - tag = Tag(self, name, attrs, self.currentTag, self.previous) - if self.previous: - self.previous.next = tag - self.previous = tag - self.pushTag(tag) - if selfClosing or self.isSelfClosingTag(name): - self.popTag() - if name in self.QUOTE_TAGS: - #print "Beginning quote (%s)" % name - self.quoteStack.append(name) - self.literal = 1 - return tag - - def unknown_endtag(self, name): - #print "End tag %s" % name - if self.quoteStack and self.quoteStack[-1] != name: - #This is not a real end tag. - #print " is not real!" % name - self.handle_data('' % name) - return - self.endData() - self._popToTag(name) - if self.quoteStack and self.quoteStack[-1] == name: - self.quoteStack.pop() - self.literal = (len(self.quoteStack) > 0) - - def handle_data(self, data): - self.currentData.append(data) - - def _toStringSubclass(self, text, subclass): - """Adds a certain piece of text to the tree as a NavigableString - subclass.""" - self.endData() - self.handle_data(text) - self.endData(subclass) - - def handle_pi(self, text): - """Handle a processing instruction as a ProcessingInstruction - object, possibly one with a %SOUP-ENCODING% slot into which an - encoding will be plugged later.""" - if text[:3] == "xml": - text = u"xml version='1.0' encoding='%SOUP-ENCODING%'" - self._toStringSubclass(text, ProcessingInstruction) - - def handle_comment(self, text): - "Handle comments as Comment objects." - self._toStringSubclass(text, Comment) - - def handle_charref(self, ref): - "Handle character references as data." - if self.convertEntities: - data = unichr(int(ref)) - else: - data = '&#%s;' % ref - self.handle_data(data) - - def handle_entityref(self, ref): - """Handle entity references as data, possibly converting known - HTML and/or XML entity references to the corresponding Unicode - characters.""" - data = None - if self.convertHTMLEntities: - try: - data = unichr(name2codepoint[ref]) - except KeyError: - pass - - if not data and self.convertXMLEntities: - data = self.XML_ENTITIES_TO_SPECIAL_CHARS.get(ref) - - if not data and self.convertHTMLEntities and \ - not self.XML_ENTITIES_TO_SPECIAL_CHARS.get(ref): - # TODO: We've got a problem here. We're told this is - # an entity reference, but it's not an XML entity - # reference or an HTML entity reference. Nonetheless, - # the logical thing to do is to pass it through as an - # unrecognized entity reference. - # - # Except: when the input is "&carol;" this function - # will be called with input "carol". When the input is - # "AT&T", this function will be called with input - # "T". We have no way of knowing whether a semicolon - # was present originally, so we don't know whether - # this is an unknown entity or just a misplaced - # ampersand. - # - # The more common case is a misplaced ampersand, so I - # escape the ampersand and omit the trailing semicolon. - data = "&%s" % ref - if not data: - # This case is different from the one above, because we - # haven't already gone through a supposedly comprehensive - # mapping of entities to Unicode characters. We might not - # have gone through any mapping at all. So the chances are - # very high that this is a real entity, and not a - # misplaced ampersand. - data = "&%s;" % ref - self.handle_data(data) - - def handle_decl(self, data): - "Handle DOCTYPEs and the like as Declaration objects." - self._toStringSubclass(data, Declaration) - - def parse_declaration(self, i): - """Treat a bogus SGML declaration as raw data. Treat a CDATA - declaration as a CData object.""" - j = None - if self.rawdata[i:i+9] == '', i) - if k == -1: - k = len(self.rawdata) - data = self.rawdata[i+9:k] - j = k+3 - self._toStringSubclass(data, CData) - else: - try: - j = SGMLParser.parse_declaration(self, i) - except SGMLParseError: - toHandle = self.rawdata[i:] - self.handle_data(toHandle) - j = i + len(toHandle) - return j - -class BeautifulSoup(BeautifulStoneSoup): - - """This parser knows the following facts about HTML: - - * Some tags have no closing tag and should be interpreted as being - closed as soon as they are encountered. - - * The text inside some tags (ie. 'script') may contain tags which - are not really part of the document and which should be parsed - as text, not tags. If you want to parse the text as tags, you can - always fetch it and parse it explicitly. - - * Tag nesting rules: - - Most tags can't be nested at all. For instance, the occurance of - a

    tag should implicitly close the previous

    tag. - -

    Para1

    Para2 - should be transformed into: -

    Para1

    Para2 - - Some tags can be nested arbitrarily. For instance, the occurance - of a

    tag should _not_ implicitly close the previous -
    tag. - - Alice said:
    Bob said:
    Blah - should NOT be transformed into: - Alice said:
    Bob said:
    Blah - - Some tags can be nested, but the nesting is reset by the - interposition of other tags. For instance, a
    , - but not close a tag in another table. - -
    BlahBlah - should be transformed into: -
    BlahBlah - but, - Blah
    Blah - should NOT be transformed into - Blah
    Blah - - Differing assumptions about tag nesting rules are a major source - of problems with the BeautifulSoup class. If BeautifulSoup is not - treating as nestable a tag your page author treats as nestable, - try ICantBelieveItsBeautifulSoup, MinimalSoup, or - BeautifulStoneSoup before writing your own subclass.""" - - def __init__(self, *args, **kwargs): - if not kwargs.has_key('smartQuotesTo'): - kwargs['smartQuotesTo'] = self.HTML_ENTITIES - kwargs['isHTML'] = True - BeautifulStoneSoup.__init__(self, *args, **kwargs) - - SELF_CLOSING_TAGS = buildTagMap(None, - ('br' , 'hr', 'input', 'img', 'meta', - 'spacer', 'link', 'frame', 'base', 'col')) - - PRESERVE_WHITESPACE_TAGS = set(['pre', 'textarea']) - - QUOTE_TAGS = {'script' : None, 'textarea' : None} - - #According to the HTML standard, each of these inline tags can - #contain another tag of the same type. Furthermore, it's common - #to actually use these tags this way. - NESTABLE_INLINE_TAGS = ('span', 'font', 'q', 'object', 'bdo', 'sub', 'sup', - 'center') - - #According to the HTML standard, these block tags can contain - #another tag of the same type. Furthermore, it's common - #to actually use these tags this way. - NESTABLE_BLOCK_TAGS = ('blockquote', 'div', 'fieldset', 'ins', 'del') - - #Lists can contain other lists, but there are restrictions. - NESTABLE_LIST_TAGS = { 'ol' : [], - 'ul' : [], - 'li' : ['ul', 'ol'], - 'dl' : [], - 'dd' : ['dl'], - 'dt' : ['dl'] } - - #Tables can contain other tables, but there are restrictions. - NESTABLE_TABLE_TAGS = {'table' : [], - 'tr' : ['table', 'tbody', 'tfoot', 'thead'], - 'td' : ['tr'], - 'th' : ['tr'], - 'thead' : ['table'], - 'tbody' : ['table'], - 'tfoot' : ['table'], - } - - NON_NESTABLE_BLOCK_TAGS = ('address', 'form', 'p', 'pre') - - #If one of these tags is encountered, all tags up to the next tag of - #this type are popped. - RESET_NESTING_TAGS = buildTagMap(None, NESTABLE_BLOCK_TAGS, 'noscript', - NON_NESTABLE_BLOCK_TAGS, - NESTABLE_LIST_TAGS, - NESTABLE_TABLE_TAGS) - - NESTABLE_TAGS = buildTagMap([], NESTABLE_INLINE_TAGS, NESTABLE_BLOCK_TAGS, - NESTABLE_LIST_TAGS, NESTABLE_TABLE_TAGS) - - # Used to detect the charset in a META tag; see start_meta - CHARSET_RE = re.compile("((^|;)\s*charset=)([^;]*)", re.M) - - def start_meta(self, attrs): - """Beautiful Soup can detect a charset included in a META tag, - try to convert the document to that charset, and re-parse the - document from the beginning.""" - httpEquiv = None - contentType = None - contentTypeIndex = None - tagNeedsEncodingSubstitution = False - - for i in range(0, len(attrs)): - key, value = attrs[i] - key = key.lower() - if key == 'http-equiv': - httpEquiv = value - elif key == 'content': - contentType = value - contentTypeIndex = i - - if httpEquiv and contentType: # It's an interesting meta tag. - match = self.CHARSET_RE.search(contentType) - if match: - if (self.declaredHTMLEncoding is not None or - self.originalEncoding == self.fromEncoding): - # An HTML encoding was sniffed while converting - # the document to Unicode, or an HTML encoding was - # sniffed during a previous pass through the - # document, or an encoding was specified - # explicitly and it worked. Rewrite the meta tag. - def rewrite(match): - return match.group(1) + "%SOUP-ENCODING%" - newAttr = self.CHARSET_RE.sub(rewrite, contentType) - attrs[contentTypeIndex] = (attrs[contentTypeIndex][0], - newAttr) - tagNeedsEncodingSubstitution = True - else: - # This is our first pass through the document. - # Go through it again with the encoding information. - newCharset = match.group(3) - if newCharset and newCharset != self.originalEncoding: - self.declaredHTMLEncoding = newCharset - self._feed(self.declaredHTMLEncoding) - raise StopParsing - pass - tag = self.unknown_starttag("meta", attrs) - if tag and tagNeedsEncodingSubstitution: - tag.containsSubstitutions = True - -class StopParsing(Exception): - pass - -class ICantBelieveItsBeautifulSoup(BeautifulSoup): - - """The BeautifulSoup class is oriented towards skipping over - common HTML errors like unclosed tags. However, sometimes it makes - errors of its own. For instance, consider this fragment: - - FooBar - - This is perfectly valid (if bizarre) HTML. However, the - BeautifulSoup class will implicitly close the first b tag when it - encounters the second 'b'. It will think the author wrote - "FooBar", and didn't close the first 'b' tag, because - there's no real-world reason to bold something that's already - bold. When it encounters '' it will close two more 'b' - tags, for a grand total of three tags closed instead of two. This - can throw off the rest of your document structure. The same is - true of a number of other tags, listed below. - - It's much more common for someone to forget to close a 'b' tag - than to actually use nested 'b' tags, and the BeautifulSoup class - handles the common case. This class handles the not-co-common - case: where you can't believe someone wrote what they did, but - it's valid HTML and BeautifulSoup screwed up by assuming it - wouldn't be.""" - - I_CANT_BELIEVE_THEYRE_NESTABLE_INLINE_TAGS = \ - ('em', 'big', 'i', 'small', 'tt', 'abbr', 'acronym', 'strong', - 'cite', 'code', 'dfn', 'kbd', 'samp', 'strong', 'var', 'b', - 'big') - - I_CANT_BELIEVE_THEYRE_NESTABLE_BLOCK_TAGS = ('noscript',) - - NESTABLE_TAGS = buildTagMap([], BeautifulSoup.NESTABLE_TAGS, - I_CANT_BELIEVE_THEYRE_NESTABLE_BLOCK_TAGS, - I_CANT_BELIEVE_THEYRE_NESTABLE_INLINE_TAGS) - -class MinimalSoup(BeautifulSoup): - """The MinimalSoup class is for parsing HTML that contains - pathologically bad markup. It makes no assumptions about tag - nesting, but it does know which tags are self-closing, that -