Skip to content

Commit 5e91da7

Browse files
yongja216codelucas
authored andcommitted
StopWordsKorean class bug (codelucas#540)
* bug fix: StopWordsKorean class fixed bug in StopWordsKorean class that original code did not find stopwords in words. StopWordsHindi class also seems like having the same problem but did not fix it. * enhancement: finding Korean stopwords Korean is composed of 'noun'+'postposition'. Postposition acts like stopwords in English. For example, to him' is '그에게'(그(he)+에게(to)) in Korean. 에게(to) is a stopword. So, we can detect stopwords mostly by checking whether the last characters in Korean words is stopwords.
1 parent 61b2bf6 commit 5e91da7

1 file changed

Lines changed: 3 additions & 2 deletions

File tree

newspaper/text.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -149,8 +149,9 @@ def get_stopword_count(self, content):
149149
c = 0
150150
for w in candidate_words:
151151
c += 1
152-
for stop_word in self.STOP_WORDS:
153-
overlapping_stopwords.append(stop_word)
152+
for s in self.STOP_WORDS:
153+
if w.endswith(s):
154+
overlapping_stopwords.append(w)
154155

155156
ws.set_word_count(c)
156157
ws.set_stopword_count(len(overlapping_stopwords))

0 commit comments

Comments
 (0)