You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Goose was originally an article extractor written in Java that has most
8
+
recently (aug2011) converted to a scala project by Gravity.com
5
9
6
-
Goose was originally an article extractor written in Java that has most recently (aug2011) converted to a scala project by Gravity.com
7
-
8
-
This is a complete rewrite in python. The aim of the software is is to take any news article or article type web page and not only extract what is the main body of the article but also all meta data and most probable image candidate.
10
+
This is a complete rewrite in python. The aim of the software is is to
11
+
take any news article or article type web page and not only extract what
12
+
is the main body of the article but also all meta data and most probable
13
+
image candidate.
9
14
10
15
Goose will try to extract the following information:
11
16
12
-
- Main text of an article
13
-
- Main image of article
14
-
- Any Youtube/Vimeo movies embedded in article (TODO)
15
-
- Meta Description
16
-
- Meta tags
17
-
17
+
- Main text of an article
18
+
- Main image of article
19
+
- Any Youtube/Vimeo movies embedded in article (TODO)
20
+
- Meta Description
21
+
- Meta tags
18
22
19
23
Originally, Goose was open sourced by Gravity.com in 2011
20
24
21
-
- Lead Programmer: Jim Plush (Gravity.com)
22
-
- Contributers: Robbie Coleman (Gravity.com)
25
+
- Lead Programmer: Jim Plush (Gravity.com)
26
+
- Contributers: Robbie Coleman (Gravity.com)
23
27
24
28
The python version was rewrite by:
25
29
26
-
- Xavier Grangier (Recrutae.com)
30
+
- Xavier Grangier (Recrutae.com)
31
+
32
+
Licensing
33
+
---------
34
+
35
+
If you find Goose useful or have issues please drop me a line, I'd love
36
+
to hear how you're using it or what features should be improved
27
37
28
-
##Licensing
29
-
If you find Goose useful or have issues please drop me a line, I'd love to hear how you're using it or what features should be improved
38
+
Goose is licensed by Gravity.com under the Apache 2.0 license, see the
39
+
LICENSE file for more details
30
40
31
-
Goose is licensed by Gravity.com under the Apache 2.0 license, see the LICENSE file for more details
There is two way to pass configuration to goose. The first one is to pass to goose a Configuration() object. The second one is to pass a configuration dict
73
+
There is two way to pass configuration to goose. The first one is to
74
+
pass to goose a Configuration() object. The second one is to pass a
75
+
configuration dict
60
76
61
-
For instance, if you want to change the userAgent used by Goose juste pass :
77
+
For instance, if you want to change the userAgent used by Goose juste
78
+
pass :
79
+
80
+
::
62
81
63
82
>>> g = Goose({'browser_user_agent': 'Mozilla'})
64
83
84
+
Switching parser : Goose can now be use with lxml html parser or lxml
85
+
soup parser. By default the html parser is used. If you want to use the
86
+
soup parser passe it in the configuration dict :
65
87
66
-
Switching parser :
67
-
Goose can now be use with lxml html parser or lxml soup parser. By default the html parser is used. If you want to use the soup parser passe it in the configuration dict :
88
+
::
68
89
69
90
>>> g = Goose({'browser_user_agent': 'Mozilla', 'parser_class':'soup'})
70
91
92
+
Goose is now language aware
93
+
---------------------------
94
+
95
+
For exemple scrapping a spanish content page with correct meta language
96
+
tags
71
97
72
-
##Goose is now language aware
73
-
For exemple scrapping a spanish content page with correct meta language tags
@@ -81,7 +106,10 @@ For exemple scrapping a spanish content page with correct meta language tags
81
106
>>> article.cleaned_text[:150]
82
107
u'Los recortes pasan factura a los pacientes. De diciembre de 2010 a junio de 2012 las listas de espera para operarse aumentaron un 125%. Hay m\xe1s ciudad'
83
108
84
-
Some pages don't have correct meta language tags, you can force it using configuration :
109
+
Some pages don't have correct meta language tags, you can force it using
@@ -90,14 +118,18 @@ Some pages don't have correct meta language tags, you can force it using configu
90
118
>>> article.cleaned_text[:150]
91
119
u'Importante golpe a la banda terrorista ETA en Francia. La Guardia Civil ha detenido en un hotel de Macon, a 70 kil\xf3metros de Lyon, a Izaskun Lesaka y '
will force as configuration will force the spanish language
121
+
Passing {'use\_meta\_language': False, 'target\_language':'es'} will
122
+
force as configuration will force the spanish language
96
123
124
+
Goose in Chinese
125
+
----------------
97
126
127
+
Some users want to use Goose for chinese content. Chinese word
128
+
segementation is way more difficult to deal with that occidental
129
+
languages. Chinese needs a dedicated StopWord analyser that need to be
130
+
passed to the config object
98
131
99
-
##Goose in Chinese
100
-
Some users want to use Goose for chinese content. Chinese word segementation is way more difficult to deal with that occidental languages. Chinese needs a dedicated StopWord analyser that need to be passed to the config object
132
+
::
101
133
102
134
>>> from goose import Goose
103
135
>>> from goose.text import StopWordsChinese
@@ -111,10 +143,13 @@ Some users want to use Goose for chinese content. Chinese word segementation is
111
143
112
144
一些亲北京阵营议员欢迎梁振英道歉,且认为应能获得香港民众接受,但这些议员也质问梁振英有
113
145
146
+
Goose in Arabic
147
+
---------------
114
148
149
+
In order to use Goose in Arabic you have to use the StopWordsArabic
150
+
class.
115
151
116
-
##Goose in Arabic
117
-
In order to use Goose in Arabic you have to use the StopWordsArabic class.
152
+
::
118
153
119
154
>>> from goose import Goose
120
155
>>> from goose.text import StopWordsArabic
@@ -124,38 +159,52 @@ In order to use Goose in Arabic you have to use the StopWordsArabic class.
124
159
>>> print article.cleaned_text[:150]
125
160
دمشق، سوريا (CNN) -- أكدت جهات سورية معارضة أن فصائل مسلحة معارضة لنظام الرئيس بشار الأسد وعلى صلة بـ"الجيش الحر" تمكنت من السيطرة على مستودعات للأسل
126
161
162
+
TODO
163
+
----
164
+
165
+
- Video extraction
127
166
128
-
##TODO
129
-
- Video extraction
167
+
Known issues
168
+
------------
130
169
131
-
##Known issues
132
-
- There is some issue with unicode URLs.
170
+
- There is some issue with unicode URLs.
133
171
134
-
##OS X 10.7 Install Instructions
172
+
OS X 10.7 Install Instructions
173
+
------------------------------
135
174
136
175
Installation Help:
137
176
138
177
1. Install python-devel if you don't have it
139
-
2. Install libjpeg
140
-
brew install libjpeg
178
+
2. Install libjpeg brew install libjpeg
179
+
180
+
3. You need to install the python imaging library. We wont be using it,
181
+
but its a dependency deep in the goose egg (fun!).
182
+
183
+
a. download
184
+
185
+
::
141
186
142
-
3. You need to install the python imaging library. We wont be using it, but its a dependency deep in the goose egg (fun!).
0 commit comments