German model lemmatizes punctuation inconsistently
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
I'm on Python 3.11 with spacy 3.7.4 and noted an [inconsistent behavior](https://stackoverflow.com/questions/79330953/lemma-of-puncutation-in-spacy/79331038) when I lemmatize my German text.
## How to reproduce the behaviour
```python
import spacy
nlp = spacy.load("de_core_news_sm")
doc = nlp("(Das ist ein Test!)")
for token in doc:
print(f"Text: '{token.text}', Lemma: '{token.lemma_}'")
```
Output:
> Text: '(', Lemma: '--'
> Text: 'Das', Lemma: 'der'
> Text: 'ist', Lemma: 'sein'
> Text: 'ein', Lemma: 'ein'
> Text: 'Test', Lemma: 'Test'
> Text: '!', Lemma: '--'
> Text: ')', Lemma: '--'
However, note the English standard model:
```python
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("(This is a test!)")
for token in doc:
print(f"Text: '{token.text}', Lemma: '{token.lemma_}'")
```
Output:
> Text: '(', Lemma: '('
> Text: 'This', Lemma: 'this'
> Text: 'is', Lemma: 'be'
> Text: 'a', Lemma: 'a'
> Text: 'test', Lemma: 'test'
> Text: '!', Lemma: '!'
> Text: ')', Lemma: ')'
On StackOverlow, the answer-giver reported the Dutch model treats punctuation as in English.
## Your Environment
* Operating System: Windows
* Python Version Used: 3.11
* spaCy Version Used: 3.7.4
* Environment Information: locally (no venv or container)
Contributor guide
Assessment
This issue has not been assessed yet.