explosion / explosion/spaCy

de_dep_news_trf uses 1990s Spelling Convention in Lemmatization

Open
#9,799 1 comment 0 reactions 0 assignees View on GitHub
feat / lemmatizer lang / de
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour
```
import spacy

nlp = spacy.load("de_dep_news_trf")
assert nlp("Du ißt Äpfel")[1].lemma_ == 'essen'
print(nlp("Du isst Äpfel")[1].lemma_)
```
This prints `isst` where `essen` would be expected.

Looks like the model just uses a lookup table which doesn't contain the 1996 changes to German spelling conventions. Same effect is observable for `frißt/frisst` as well.

## Your Environment
de-dep-news-trf @ https://github.com/explosion/spacy-models/releases/download/de_dep_news_trf-3.2.0/de_dep_news_trf-3.2.0-py3-none-any.whl
spacy==3.2.0
spacy-alignments==0.8.4
spacy-legacy==3.0.8
spacy-loggers==1.0.1
spacy-transformers==1.1.2

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.