Brazilian portuguese support

[julianofischer]

I would like to use newspaper in brazilian sites. I tested it but unsuccessfully.
The articles are not extracted correctly.

newspaper is internationalized and I am using it in a wrong way or can we make it able to brazilian portuguese?

Thank you.

[codelucas]

I just tested newspaper on some spanish websites and it works, however you need to toggle the configuration settings. (I know they speak Portuguese in Brazil but the two languages are at least similar.)

Newspaper works like this:

If you don't toggle any of the configs, our extractor tries to find the language of the article with meta tags. If isn't capable of doing so (most of the time the meta tags are empty for languages), it defaults on english, or 'en'. This is probably happening to you.

Here is a working example of extracting from an article where you know the language of the article is spanish:

>>> from newspaper import Article, Config
>>> config = Config()
>>> config.use_meta_language = False    # it's a common mistake to forget to toggle this section
>>> config.target_language = 'es'              # spanish
>>> url = 'http://latino.foxnews.com/latino/espanol/2013/12/29/el-volcan-chaparrastique-lanza-gases-y-cenizas-en-la-zona-oriental-de-el/'

>>> article = Article(url=url, config=config) # note that we are using named arguments, we need to! 
>>> article.download()
>>> article.parse()

>>> article.text

u'El volc\xe1n Chaparrastique lanz\xf3 el domingo gases y cenizas en la zona oriental de El Salvador, por lo que las autoridades decidieron evacuar a los pobladores de los alrededores.\n\nLa actividad se registr\xf3 a las 10.30 a.m. y consisti\xf3 "en una columna eruptiva de gases y cenizas, la cual se elev\xf3 aproximadamente a unos 5 kil\xf3metros", inform\xf3 el ministerio de Medio Ambiente y Recursos Naturales.\n\nAlgunas poblaciones cercanas han comenzado a evacuar la zona por precauci\xf3n, pero no se reportan v\xedctimas, se\xf1al\xf3 la Radio Nacional de El Salvador.\n\nEl vocero de la Cruz Roja en el pa\xeds, Carlos L\xf3pez Mendoza, dijo a The Associated Press que su personal "ya est\xe1 en la zona, Protecci\xf3n Civil orden\xf3 evacuar a la gente a tres kil\xf3metros del volc\xe1n, pero no tenemos reportes de v\xedctimas".\n\n"Nuestra gente dice que se escuch\xf3 una explosi\xf3n bastante fuerte, y luego las nubes negras se han corrido por toda la zona oriental del pa\xeds, especialmente en San Miguel, Usulut\xe1n y en la Uni\xf3n", agreg\xf3.\n\nEl volc\xe1n Chaparrasti est\xe1 ubicado en el departamento de San Miguel, a 135 kil\xf3metros al este de la capital y tiene una altura de 2.129 metros sobre el nivel del mar. La \xfaltima vez que hizo erupci\xf3n fue el 29 de diciembre de 1976.'

We got the entire article.

As long as the language is supported by a stopword set, newspaper will be able to extract the article. View that set here:
https://github.com/codelucas/newspaper/tree/master/newspaper/resources/text

Contribute when you can!

Also, if you know that an entire news source is in a certain language, you may pass a config object to the entire news source and that config object will get passed on to all of the children articles.

>>> from newspaper import Source, Article, Config

>>> config = Config()
>>> config.use_meta_language = False   
>>> config.target_language = 'fr' # french

>>> url = 'http://www.lemonde.fr/'

>>> source = Source(url=url, config=config)
>>> source.build()

>>> source.articles[0].download()
>>> source.articles[0].parse()
>>> source.articles[0].text

Thanks.

[codelucas]

Actually, this raises a good issue of the fact that it is so complicated to use different languages.

After you respond, i'm gonna close this issue and open up a new one to make language toggling more simplistic. We also have arabic and chinese support but there is even an extra step to toggle non-occidental languages.

[julianofischer]

Hi codelucas,
thank you very much for your reply.

Is there a way to specify the language to the newspaper.build(url) method?
By the way, this project is amazing, congratulations!
I will try to contribute any way I can.

[julianofischer]

Sorry, I read the second example and realized =$
Thank you again.

[julianofischer]

The "engine" is entirely based on the stopword set?
There is no set for brazilian portuguese (pt-br) =(
How can I contribute with this set?

[codelucas]

A Stopword set is required for full text extraction.

The entire internationalization portion still needs a big overhaul, it works as of now but it is not easy to use at all.
In 2-3 days changes and test cases will be pushed.

For now, you can try spanish? 'es', tell me how it works out. I know Spanish and Portuguese are at least similar.

I'm gonna need to set up the testing architecture so it is easier to contribute but for now, the Portuguese Stopword set is already online:

http://snowball.tartarus.org/algorithms/portuguese/stop.txt

We need to make that into the format of a text file delimited by newlines first.
Then run test cases on actual brazilian articles to make sure it works.

[julianofischer]

I have tested with 'es' and seemingly to works fine.
If you need, I will try to make the stopword file tomorrow.

I have one more question.
I tested with the link "www.globo.com", a well-known news site in Brazil and it works.
But with "globoesporte.globo.com", the Globo.com section dedicated to sports news, seems not to work.

What is the easiest way to find out why it does not work?

[codelucas]

Let me try to re-type your problem to you so I am sure I understand what you are saying.

When you try to build a newspaper for globo.com, your categories are --> noticias, esportes, entrteinmto, technologia, etc. However, when you run the same command on globoesporte.globo.com, instead of getting --> futbol, mma, F1 racing, you are getting nothing?

[codelucas]

OK, it is very good you mentioned this. This is a serious error. So, I ran newspaper.build(...) on globoesporte.globo and instead of giving me the categories for that specific subdomain (f1, mma, futbol), I got the categories for the globo domain (esportes, entrtainmo, technologia, ..).