Chinese segmentation is not separating quotation marks correctly
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## How to reproduce the behaviour
Run the following code:
```
import spacy
nlp = spacy.load('zh_core_web_trf')
parsed = nlp("""故事不是用"说"故事或"讲"故事
为什么我们要用"聊"故事呢?""")
for token in parsed:
print(token.text, token.pos_)
```
which produces the following output:
```
故事 NOUN
不是 CCONJ
用" VERB
说 VERB
" PUNCT
故事 NOUN
或 CCONJ
"讲 VERB
" PUNCT
故事 NOUN
SPACE
为什么 ADV
我们 PRON
要 VERB
用 VERB
"聊 VERB
" PUNCT
故事 NOUN
呢 PART
? PUNCT
```
Note that the quotation mark is not always separated from the word correctly, such as in `用"`, `"讲` and `"聊`. The behavior is not even internally consistent: sometimes the opening quote is separated, sometimes it is not, sometimes the ending quote is separated, and sometimes it is not.
## Info about spaCy
- **spaCy version:** 3.7.5
- **Platform:** Windows-11-10.0.22631-SP0
- **Python version:** 3.12.4
- **Pipelines:** en_core_web_sm (3.7.1), en_core_web_trf (3.7.3), es_dep_news_trf (3.7.2), ru_core_news_lg (3.7.0), uk_core_news_trf (3.7.2), zh_core_web_trf (3.7.2)
Contributor guide
Assessment
This issue has not been assessed yet.