baidu / baidu/lac

词典中加入单词对分词影响的一个bug

Open
#222 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
4k
Forks
592
PR merge metrics
No merged PRs in 30d

Description

在字典加入的单词中以已有单词的后一个字开头的新单词,与已有单词放一起分词时会有问题!

例如:
马达加斯加普通话版 (字典中加入 马达加斯加/ip 加普/star 普通话版/other)

分词结果为: 马达加斯 加 普通话版

张继科比赛 (字典中加入 张继科/star 科比/star)

分词结果:张继 科比 赛

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the two examples with the stated dictionary entries and inspect the dictionary-based word-segmentation path. Compare the actual boundaries with the expected words “马达加斯加”, “加普”, “张继科” and “科比”; done means both examples segment into the dictionary words without truncation or unintended merging.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.