explosion / explosion/spaCy

German model lemmatizes punctuation inconsistently

Open
#13,729 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

I'm on Python 3.11 with spacy 3.7.4 and noted an [inconsistent behavior](https://stackoverflow.com/questions/79330953/lemma-of-puncutation-in-spacy/79331038) when I lemmatize my German text.

## How to reproduce the behaviour
```python
import spacy

nlp = spacy.load("de_core_news_sm")
doc = nlp("(Das ist ein Test!)")
for token in doc:
print(f"Text: '{token.text}', Lemma: '{token.lemma_}'")
```

Output:

> Text: '(', Lemma: '--'
> Text: 'Das', Lemma: 'der'
> Text: 'ist', Lemma: 'sein'
> Text: 'ein', Lemma: 'ein'
> Text: 'Test', Lemma: 'Test'
> Text: '!', Lemma: '--'
> Text: ')', Lemma: '--'

However, note the English standard model:

```python
import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("(This is a test!)")
for token in doc:
print(f"Text: '{token.text}', Lemma: '{token.lemma_}'")
```

Output:

> Text: '(', Lemma: '('
> Text: 'This', Lemma: 'this'
> Text: 'is', Lemma: 'be'
> Text: 'a', Lemma: 'a'
> Text: 'test', Lemma: 'test'
> Text: '!', Lemma: '!'
> Text: ')', Lemma: ')'

On StackOverlow, the answer-giver reported the Dutch model treats punctuation as in English.

## Your Environment
* Operating System: Windows
* Python Version Used: 3.11
* spaCy Version Used: 3.7.4
* Environment Information: locally (no venv or container)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.