UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 11: ordinal not in range(128)

[timini]

File "/home/tim/Workspace/Development/hacks/pressmonitor/pressmon/articles/management/commands/collect_articles.py", line 27, in handle
article.nlp()
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/newspaper/article.py", line 307, in nlp
summary_sents = nlp.summarize(title=self.title, text=self.text)
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/newspaper/nlp.py", line 34, in summarize
sentences = split_sentences(text)
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/newspaper/nlp.py", line 146, in split_sentences
sentences = tokenizer.tokenize(text)
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/nltk/tokenize/punkt.py", line 1270, in tokenize
return list(self.sentences_from_text(text, realign_boundaries))
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/nltk/tokenize/punkt.py", line 1318, in sentences_from_text
return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/nltk/tokenize/punkt.py", line 1309, in span_tokenize
return [(sl.start, sl.stop) for sl in slices]
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/nltk/tokenize/punkt.py", line 1348, in _realign_boundaries
for sl1, sl2 in _pair_iter(slices):
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/nltk/tokenize/punkt.py", line 355, in _pair_iter
for el in it:
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/nltk/tokenize/punkt.py", line 1324, in _slices_from_text
if self.text_contains_sentbreak(context):
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/nltk/tokenize/punkt.py", line 1369, in text_contains_sentbreak
for t in self._annotate_tokens(self._tokenize_words(text)):
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/nltk/tokenize/punkt.py", line 1504, in _annotate_second_pass
for t1, t2 in _pair_iter(tokens):
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/nltk/tokenize/punkt.py", line 354, in _pair_iter
prev = next(it)
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/nltk/tokenize/punkt.py", line 621, in _annotate_first_pass
for aug_tok in tokens:
File "/home/tim/.virtualenvs/pressmon/lib/python2.7/site-packages/nltk/tokenize/punkt.py", line 586, in _tokenize_words
for line in plaintext.split('\n'):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 11: ordinal not in range(128)

[timini]

the text for the article is unicode:

u'Towels and sweat-ridden bedclothes remained for two days in the Dallas apartment where an undiagnosed Ebola sufferer was staying because health officials in Texas struggled to find a waste management company willing to accept them.\n\nA company with experience of dealing with medical waste was finally due to clean the apartment on Thursday afternoon, but only after the patient\u2019s girlfriend told CNN that the materials had not been disposed of.\n\nHealth officials obtained an order to force the residents of the apartment to stay at home, after admitting they had not complied with a voluntary request to do so on Wednesday. The order applies to four \u201cclose\u201d family members, including a woman believed to be the girlfriend of the patient, Thomas Duncan, as well as her child and two nephews in their 20s. Local police have been stationed at the apartment complex to ensure compliance.\n\nState health officials estimate the number of people who came into direct contact with Duncan as being between 12 and 18, including five school-age children. They are being actively monitored, a procedure that includes taking the person\u2019s temperature twice a day. They are also working from a list of 100 potential or possible further contacts.\n\n\u201cOut of an abundance of caution, we\u2019re starting with this very wide net, including people who have had even brief encounters with the patient or the patient\u2019s home,\u201d said Carrie Williams, a spokeswoman with the Texas department of state health services.\n\nOfficials in Liberia have threatened to prosecute Duncan for failing to disclose on a health form when he left the country late last month that he had come into contact with someone suffering from Ebola.\n\nBinyah Kesselly, chairman of the Liberia Airport Authority,told the Associated Press that Duncan ticked \u201cno\u201d on a questionnaire that asked if he he had come in contact with anyone who was infected with or had died from Ebola. \n\n\n\nThe New York Times reported that while in Liberia, Duncan helped his landlords transport their pregnant daughter, who showed symptoms of the disease, to hospital. The young woman later died.\n\nAt a rowdy press conference in Dallas, officials were repeatedly pressed by reporters on apparent missteps in their response to the Ebola diagnosis, the first outside Africa, where the disease has caused more than 3,000 deaths since the outbreak began in March. On Wednesday, hospital chiefs admitted they sent Duncan home with a course of antibiotics even though he told a nurse he had travelled from Liberia. The nurse did not share the information more widely with the medical team assessing Duncan.\n\nJudge Clay Jenkins, Dallas county\u2019s emergency management director, said officials had now found a suitable contractor to clean the home where Duncan was staying. \u201cWe have some hygiene issues that we are addressing in that apartment,\u201d Jenkins said. He added that the contractors would be at the apartment \u201cas soon as possible\u201d.\n\nJenkins said the sheets and towels, along with Duncan\u2019s belongings, were in sealed plastic bags.\n\n\u201cThe house conditions need to be improved,\u201d Dr David Lakey, the Texas health commissioner, acknowledged during a separate conference call with reporters on Thursday.\n\nLakey said local and county officials had experienced difficulties in securing a contractor to remove the waste from the apartment and clean it. \u201cThere has been a little bit of hesitancy for entities to want to do that,\u201d Lakey said.\n\nHe added on Twitter later that his agency was arranging for food and groceries to be delivered. He said that Duncan, who is being treated at a hospital less than a mile from the apartment, has a telephone and is able to make calls, and that there is \u201cno intention to put other individuals under a control order\u201d.\n\nDuncan was staying with family in an apartment complex in north-east Dallas, eight miles from downtown. After being sent home from hospital on Friday, he fell seriously ill on Sunday and an ambulance was called to take him from the residence to hospital.\n\nSally Nuran, the manager of the apartment complex, told reporters gathered outside the front entrance on Thursday that one woman and two children were registered as living at the property.\n\nNuran said that she is confident that Duncan did not closely interact with anyone outside when he became infectious following his arrival in Dallas from Liberia on 20 September, though a man who lives on the complex said on Wednesday that he had seen him stagger outside and vomit on the ground as the ambulance arrived.\n\n\u201cThe person did not get in contact with anyone on the property,\u201d she said. \u201cThere is no fear about the common areas, the guy wasn\u2019t here for that long.\u201d\n\nNuran said that she was notified by health officials from various agencies about the possible diagnosis on Monday night. The information was publicly released on Tuesday afternoon. The family had been expected to move out on 30 September, Nuran said, because their lease was up.\n\nNow they are under a 21-day quarantine by order of Texas state and county health officials, banning them from leaving the flat without approval until at least 19 October, when the incubation period will be over, and mandating them to report any symptoms and submit to health tests if deemed necessary. Failure to comply could result in criminal charges.\n\nLocal government officials said that the complex has 300 units and about 25,000 people live in the densely populated, multi-ethnic district known as Vickery Meadow.\n\nAt midday on Thursday, a child peeked out from behind a red diamond-pattered curtain in one of the apartments while at ground level a team of three contractors \u2013 none wearing any sort of protective clothing \u2013 power-washed the front porch. A stroller stood at the bottom of a staircase.\n\nEarlier, a representative of one of the agencies who issued the control order said that arranging clean bedding was the responsibility of the family \u2013 despite the ban on them leaving their home. \u201cThe individuals, it\u2019s up to them \u2026 to care for the household,\u201d Erikka Neroes of Dallas County health and human services told the Guardian. \u201cOur science tells us, according to CDC, that Ebola virus germs can be killed with soap and water \u2026 Dallas County has not been involved in a disinfection process.\u201d\n\nEarlier in the day, angrily brandishing an Ebola fact sheet, David Mbusa strode out of the complex and told reporters that he was completely unaware that a fellow resident had the virus until this morning.\n\nMbusa said he is a night shift worker from South Sudan and has lived in the Dallas area for about 15 years. He said that he woke up this morning to find a one-page general information sheet about the virus on his door and was confused by the police presence in the complex and asked a small boy what was happening.\n\n\u201cI don\u2019t know anything about Ebola,\u201d he said. \u201cThis thing is happening in west Africa.\u201d After reporters updated him, he said that he was \u201cupset completely, I\u2019m not sure what\u2019s going on \u2026 Me and everybody who lives here should be screened, that hasn\u2019t happened \u2026 whoever lives here should stay inside.\u201d\n\nNuran said that she had arranged for flyers to be posted on every apartment door on Wednesday evening but the variety of different languages spoken by residents made communication difficult and it took time to translate the information. She said that the exterior of the whole property would be cleaned and that representatives from the CDC and other government departments were in regular contact and making visits.\n\nJennifer Gates, a local council member, said that \u201ceight to 10\u201d CDC officials had visited the complex on Wednesday but acknowledged that some people are \u201cscared\u201d and \u201cconfused\u201d and that many Meadow residents speak little or no English.\n\nSeveral schools are in the area and parents have expressed concern about their children\u2019s safety. \u201cThe school is a safe place, please continue to send your kids to school,\u201d she said. Dallas Independent school district said in a statement that its schools are following a normal schedule but nurses are on twice-daily rounds visiting schools attended by five children who may have come into contact with Duncan. The five children are now being schooled at home.\n\nA spokesman for the Dallas Independent School District said attendance was down by 10% across the five affected campuses on Thursday, with higher absences at the elementary schools.\n\nOfficial reassurances do not seem to be carrying much weight. \u201cNobody really knows what\u2019s going on,\u201d said Chris Cooper, a local resident who was driving past the complex. \u201cHell yeah, I\u2019m very worried.\u201d'

[timini]

Am I using the nlp method correctly?

[gmgreg]

Perhaps in nlp.py lines 28-31 should change to encode in ascii instead of utf-8 for NLTK uses:

if isinstance(title, unicode):
    title = title.encode('ascii', 'ignore')
if isinstance(text, unicode):
    text = text.encode('ascii', 'ignore')

I'm no expert but this worked for me...

[codelucas]

@gmgreg Good suggestion.

I'm closing this issue as tons of changes have happened since, newspaper has also been ported to python3. This encoding issue should not exist anymore.