Not extracting UL LI text

[raistie]

Bulleted points in articles are not extracted and are totally missing from the extracted text

[codelucas]

Read Grangier's post in:
grangier/python-goose#103

He makes a good point on the fact that cleaning li and ul elements is a tradeoff, but there are definitely work arounds. I'll think about this and let you know in the future.

[grangier]

hello @codelucas ... juste discovering your project.
Seems you used more than a lot of python-goose code here, without leaving credit header to complete goose file.

Moreover, it keeping git history log in the files you use is not a priority for you.

[grangier]

@raistie as newspaper use python-goose extraction same bug ...

[codelucas]

Hey @grangier, thanks for porting your original goose project!. Yeah, a lot of goose was used, especially the extractor. How much of the header would you like me to include on the files?

The entire:

This is a python port of "Goose" orignialy licensed to Gravity.com
under one or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.

Python port was written by Xavier Grangier for Recrutae

Gravity.com licenses this file
to you under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

?

Moreover, I created a new repo for this instead of pushing or forking because I
still made pretty big changes + wanted to make more in the future. But goose's
core extraction techniques are used.

By git history log do you mean the HISTORY.md file?

[jeffnappi]

Hey guys, just thought I'd chime in on this conversation as it is quite relevant to me. I'm currently using newspaper as well as python-goose and I've had to trace a bug back through both projects and submit independent bug reports. From what I've seen, newspaper might be better served requiring the python-goose module and extending it rather than replacing it.

@grangier has been very responsive to pull requests and I would think that he would accept all tested and reasonable upstream improvements.

@codelucas I understand that your concept has been to pull together these disparate libraries together to make them simple to use a la requests. Though if you look at requests you'll see that it's urllib3 at the core and does not simply copy that code base.

Regardless if we all work together on article extraction for Python, we'll have better results than maintaining independent forks.

[jeffnappi]

@codelucas bundling python-goose as requests does urllib3 might make sense: see https://github.com/kennethreitz/requests/tree/master/requests/packages

[codelucas]

@jeffnappi - Thanks for the fix on that negative node bug btw :p

Good point. I did not package Goose in the code (or make it a pip dependency) because I thought i'd make more changes to it instead of just interfacing with the functions.
I'll think about it, but i'm leaning more towards just making a new branch and packaging goose in on that one.

[jeffnappi]

@codelucas no worries :) I definitely appreciate the work you've done on newspaper - it's great!

[codelucas]

Thanks @jeffnappi! Hopefully more updates will come after school ends.

And @grangier, i'll add the original goose license headers to all containing files this weekend.