Stanza - A Python NLP Library for Many Human Languages

Sun, 12 Apr 2020 00:00:00 +0000

I tested out Stanza. English tokenizer and definately works. I ran quick test with Japanese lang and output was somewhat unexpected.

import stanza

# japanese "ja", for english model "en"

stanza.download("ja")
nlp = stanza.Pipeline("ja")
doc = nlp("皆さんおはようございます！　ご機嫌いかがですか？")

for i, sentence in enumerate(doc.sentences):
 print(f"===== Sentence {i+1} tokens =====")
 print(*[f"word: {word.text}\t upos: {word.upos} xpos: {word.xpos}" for word in sentence.words], sep="\n")

The output is:

Nlp on the art of simplicity

Stanza - A Python NLP Library for Many Human Languages