explosion / explosion/spaCy

Russian pos tagging/lemmatization/morphological analysis fails with diacritics

Open
#12,530 5 comments 0 reactions 0 assignees Claimed by @adrianeboyd View on GitHub
lang / ru lang / uk
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

It seems that while there is support for tokenization with diacritics in spaCy, the project doesn't lemmatize/morph/pos tag correctly when they are used.

## How to reproduce the behaviour
```
import ru_core_news_lg
nlp = ru_core_news_lg.load()
doc = nlp('Я ви́жу му́жа и жену́')
print(doc[-1].pos_) # PROPN (incorrect. just a noun)
print(doc[-1].lemma_) # жену́ (incorrect. should be жена)
print(doc[-1].morph) # nothing is printed which is obviously incorrect
```

if changed to remove the diacritics all is well
```
from spacy.lang.char_classes import COMBINING_DIACRITICS
diacritics_re = re.compile(f'[{COMBINING_DIACRITICS}]')
doc = nlp(diacritics_re.sub('', 'Я ви́жу му́жа и жену́'))

print(doc[-1].pos_) # NOUN
print(doc[-1].lemma_) # жена
print(doc[-1].morph) # Animacy=Anim|Case=Acc|Gender=Fem|Number=Sing
```

## pymorphy3/pymorphy2 doesn't handle diacritics
it seems pymorphy3/2 doesn't handle diacritics, so perhaps before `parse` is called, diacritics should be removed.
```
diacritics_re = re.compile(f'[{COMBINING_DIACRITICS}]')
text = diacritics_re.sub('', token.text)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.