StopWordsKorean class bug

[yongja216]

Fixed Bug in StopWordKorean class

fixed bug in StopWordsKorean class that original code did not find stopwords in words.
StopWordsHindi class also seems like having the same problem but did not fix it.

Enhancement in StopWordKorean class

Korean is composed of 'noun'+'postposition'.
Postposition acts like stopwords in English.
For example, to him' is '그에게'(그(he)+에게(to)) in Korean. 에게(to) is a #stopword.
So, we can detect stopwords mostly by checking whether the last characters in Korean words are stopwords.

[codelucas]

Very interesting, thanks for this fix @yongja216! 💯 👍 We also need to create a bug ticket for the hindi fix too (i've actually been trying to learning korean so this is super helpful 😄 )

However, is it always the case the postposition is after the noun instead of before? Some examples in http://mylanguages.org/korean_prepositions.php show a pre-position (but maybe i am mis-understanding korean word segmentation, such as:
(but) I'm busy - (하지만) 나는 바뻐요. In this case but comes before

If there is a chance it can come before maybe an endswith + startswith check both together will be the best. Let me know!

[yongja216]

@codelucas
(but) I'm busy - (하지만) 나는 바뻐요.
Above sentence will be split by candidate_words to '하지만(but)', '나는(I'm)', '바빠요(busy)'. '하지만' is always used alone, not combined with nouns. So endswith can check 하지만(but). Korean prepositions in this list http://mylanguages.org/korean_prepositions.php are actually not preposition(instead, come at the end of nouns) or used alone(like '하지만(but)').

There are also other stopwords in '나는' and '바빠요'; 나(I)+는(am; stopword), 바빠(busy)+요(word used at the end of sentence in formal polite expression; stopword). These two stopwords can be checked by endswith also. However these words (~는, ~요) are not stopwords if they come at the first position.(these characters don't come at the first position as stopword) Maybe most words except '이' in stopword_ko.txt are used at the end of words.

I would be grad that it could help you :)

[codelucas]

Very helpful! Approving and merging @yongja216

[codelucas]

I will try to create a new version release soon so this can make it in