explosion / explosion/spaCy

Chinese segmentation is not separating quotation marks correctly

Open
#13,596 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour
Run the following code:

```
import spacy
nlp = spacy.load('zh_core_web_trf')
parsed = nlp("""故事不是用"说"故事或"讲"故事
为什么我们要用"聊"故事呢?""")
for token in parsed:
print(token.text, token.pos_)
```

which produces the following output:

```
故事 NOUN
不是 CCONJ
用" VERB
说 VERB
" PUNCT
故事 NOUN
或 CCONJ
"讲 VERB
" PUNCT
故事 NOUN

SPACE
为什么 ADV
我们 PRON
要 VERB
用 VERB
"聊 VERB
" PUNCT
故事 NOUN
呢 PART
? PUNCT
```

Note that the quotation mark is not always separated from the word correctly, such as in `用"`, `"讲` and `"聊`. The behavior is not even internally consistent: sometimes the opening quote is separated, sometimes it is not, sometimes the ending quote is separated, and sometimes it is not.

## Info about spaCy

- **spaCy version:** 3.7.5
- **Platform:** Windows-11-10.0.22631-SP0
- **Python version:** 3.12.4
- **Pipelines:** en_core_web_sm (3.7.1), en_core_web_trf (3.7.3), es_dep_news_trf (3.7.2), ru_core_news_lg (3.7.0), uk_core_news_trf (3.7.2), zh_core_web_trf (3.7.2)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.