Problem in Brazilian sites

[julianofischer]

I got problems using the newspaper in Brazilian sites.
Following is an example:

import newspaper

info = newspaper.build('http://globoesporte.globo.com/futebol/times/sao-paulo')
len(info.artices)

It returned only 3 articles.

Sorry if I am using it wrongly.

[codelucas]

Ah, you are using the library incorrectly. But no worries, this is probably a design flaw because so many people are getting this wrong. Newspaper, by default, caches all articles which have been downloaded from any Source. Any article which has already been downloaded will never be downloaded again, unless you turn the memoize_articles setting to False.

Here is the code exactly from my computer and some examples. BTW please refer to the docs for specifics.

>>> import newspaper

>>> info = newspaper.build('http://globoesporte.globo.com/futebol/times/sao-paulo')
>>> len(info.articles) 
221 

So now lets run the above command again and see what happens:

>>> info = newspaper.build('http://globoesporte.globo.com/futebol/times/sao-paulo')
>>> len(info.articles) 
0    

Looks like they got cached.

Let's wait 20 minutes before running it again...
...

>>> info = newspaper.build('http://globoesporte.globo.com/futebol/times/sao-paulo')
>>> len(info.articles) 
12

This means that globo has published 12 new articles!
However, you can disable this setting completely.

>>> info = newspaper.build('http://globoesporte.globo.com/futebol/times/sao-paulo', memoize_articles=False)
>>> len(info.articles)
221

>>> info.size()
221

^ Please note that .size() returns the number of articles, it's easier than calling len(info.articles)!

Also, even though auto language detection is enabled, it is still better to pass in a language if you know the language of a news source, just in case newspaper messes up.

So,

info = newspaper.build('http://globoesporte.globo.com/futebol/times/sao-paulo', language='pt')

[codelucas]

Please let me know if the problem is solved so I can close the issue. Also, should memoize_articles be set to False on default? I think we need a poll, because a lot of people are getting confused.

[julianofischer]

Hi Lucas,
it worked. Thank you!
I think inserting these code snippets in documentation will mitigate the doubts and errors. They are very instructive.

However, there are many articles with no title setted and several highlighted articles did not appear, a little precision problem, I think.

Thanks again by the quick attention.

[codelucas]

Hmm this may be another usage issue! Sorry haha. So for the titles... They are only fully set after u call the parse() method, which of course requires the download() method to be called.

Have u called download() and parse() on all of the articles of info?

>>> for a in info.articles:
>>>   a.download()

>>> for a in info.articles:
>>>   a.parse()

You may notice that SOME titles are set prior to having called parse(). This is because newspaper makes educated guesses on where the titles are early!

[codelucas]

I just tested it on my computer, the titles all work 100% after parse().

>>> for a in info.articles:
...   a.download()
... 
>>> for a in info.articles:
...   a.parse()
... 
>>> for a in info.articles:
...   print 'title', a.title
... 
title Toquinho estreia no WSOF em março contra Jon Fitch, Burkman ou Harris - combate
title Um ano após câncer, Guilherme Leme ainda se recupera da doença
title Volkswagen divulga primeira foto oficial do Up! brasileiro; compare
title Google Glass ganha app de exercícios com realidade aumentada
title Vídeo mostra momento em que advogado foragido é preso na PB
title Briga de casal em avião obriga piloto a pousar em Salvador, diz PF
title Rivais? Marlon e William saem no tapa
title Funcionários pedem que travesti não use banheiro feminino em shopping
title Jogador faz recuo ilegal, e juiz tem que ensinar regra a todos em campo
title Veja o guia do IPVA 2014
title Estreia de Minotauro
title Companhia aérea perde mala e ator Rafael Cardoso registra no Instagram - Marie Claire
title BMW faz recall de 56 unidades da moto F 800 S no Brasil
title Advogado fugitivo é preso em pousada na orla de João Pessoa
title Revelação em 'Além do Horizonte', Marina Palha diz: 'Deixei as frescuras de lado'
title São Paulo 5 x 1 Ypiranga-SP, 09/01/1955 -
title Bernardinho garante que não será candidato ao governo do Rio
title São Paulo muda olhar sobre o Paulistão, agora meta do clube
title Vale tudo? Longe disso. Conheça as regras usadas pelo UFC - combate
title Sete suspeitos de matar ex-miss em assalto são presos na Venezuela
title Portela promete desfile com a maior escultura que já cruzou a Sapucaí
...