词典中加入单词对分词影响的一个bug
Open
- Dominant language
- C++
- Stars
- 4k
- Forks
- 592
- PR merge metrics
- No merged PRs in 30d
Description
在字典加入的单词中以已有单词的后一个字开头的新单词,与已有单词放一起分词时会有问题!
例如:
马达加斯加普通话版 (字典中加入 马达加斯加/ip 加普/star 普通话版/other)
分词结果为: 马达加斯 加 普通话版
张继科比赛 (字典中加入 张继科/star 科比/star)
分词结果:张继 科比 赛
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the two examples with the stated dictionary entries and inspect the dictionary-based word-segmentation path. Compare the actual boundaries with the expected words “马达加斯加”, “加普”, “张继科” and “科比”; done means both examples segment into the dictionary words without truncation or unintended merging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100