explosion / explosion/spaCy

German tokenizer splits gender-inclusive colon forms (Kund:innen)

Open
#14,010 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour

The German infix rules split a colon between letters, which breaks
gender-inclusive forms, standard orthography in contemporary German
business and public-sector text, before tagging or NER ever see the word:

```python
import spacy

nlp = spacy.load("de_core_news_lg") # tokenizer behavior; any de pipeline
print([t.text for t in nlp("Unsere Kund:innen und Mitarbeiter:innen sind zufrieden.")])
# ['Unsere', 'Kund', ':', 'innen', 'und', 'Mitarbeiter', ':', 'innen', 'sind', 'zufrieden', '.']
print([t.text for t in nlp("Wir suchen eine:n Ärzt:in für unser Team.")])
# ['Wir', 'suchen', 'eine', ':', 'n', 'Ärzt', ':', 'in', 'für', 'unser', 'Team', '.']
```

Expected: `Kund:innen`, `Mitarbeiter:innen`, `eine:n`, `Ärzt:in` as single
tokens. For comparison, the star and interpunct variants already stay
whole (`Kolleg*innen`, `Expert·innen`), only the colon splits, via this
infix in `spacy/lang/de/punctuation.py`:

```python
r"(?<=[{a}])[:<>=](?=[{a}])".format(a=ALPHA)
```

The gender colon is the most common separator in German inclusive writing
(the INCLUSIFY benchmark, [arXiv:2212.02564](https://arxiv.org/abs/2212.02564), documents frequency across sources). Every downstream component degrades on the split: the noun is tagged in pieces, NER never sees the name-like whole, and lemmas are
computed for the fragments. We created an [inclusive language text checker](https://github.com/witty-works/nlp_api) and patch this infix; we'd like to upstream the behavior.

Two possible fixes, happy to submit a PR for either:

1. Conservative: exempt the noun gender endings from the colon infix -
`r"(?<=[{a}]):(?!in(nen)?\b)(?=[{a}])"` - which keeps `Kund:in` and
`Kund:innen` whole. Article forms (`eine:n`, `jede:r`) would still
split; covering them too means exempting short lowercase tails, e.g.
`r"(?<=[{a}]):(?![a-zäöüß]{1,5}\b)(?=[{a}])"`, at the cost of no longer
splitting all-lowercase typos like `wort:wort`.
2. A tokenizer config option, if changing the default is unwanted.

## Your Environment

* Operating System: macOS-26.6.1-arm64-arm-64bit
* Python Version Used: 3.12.9
* spaCy Version Used: 3.8.14
* Environment Information: de_core_news_lg 3.8.0 (behavior identical on
3.7.x; long-standing default, not a regression)

## Notes

- The original solution was developed without AI assistance; however, this ticket was written with the help of Claude. I was unable to find an AI policy for this project.
- Our [production patch](https://github.com/witty-works/nlp_api/blob/dev/app/model.py#L350) keeps `<>=` splitting and drops only the colon.
- I have another ticket in the pipeline related to ADJA mistagging of colon forms, which is only observable once they survive tokenization.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the German examples with the tokenizer, then inspect the infix rule in spacy/lang/de/punctuation.py. Compare the conservative ending exemption with the configurable alternative, and consider the existing production patch for behavior; done means the listed colon forms remain single tokens without losing the existing <> splitting behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.