'seg'和'lac'分词结果不一致
- Dominant language
- C++
- Stars
- 4k
- Forks
- 592
- PR merge metrics
- No merged PRs in 30d
Description
### 问题:
pip默认安装,发现`lac_model`,`seg_model`,`rank_model`下`word.dic`字典文件不同,导致分词结果不一致。
`lac_model\conf`下`word.dic`,746KB,58223行,包含多字词语;
`rank_model\conf`下`word.dic`,746KB,58223行,包含多字词语,和`lac_model`相同;
`seg_model\conf`下`word.dic`,71KB,8223行,只包含单字。
```
text = "“没有什么比这场疫情下的生与死更能体现美国的肤色差异了”。"
lac_seg = LAC(mode='seg')
seg_result = lac_seg.run(text)
lac_lac = LAC(mode='lac')
lac_result = lac_lac.run(text)
lac_rank = LAC(mode='rank')
rank_result = lac_rank.run(text)
```
### **结果:**
seg_result = ['“', '没有', '什么', '比', '这', '场', '疫情', '下', '的', '生与死', '更', '能', '体现', '美国', '的', **'肤色'**, **'差异'**, '了', '”', '。']
lac_result = [['“', '没有', '什么', '比', '这场', '疫情', '下', '的', '生与死', '更', '能', '体现', '美国', '的', **_'肤'_**, **_'色差异'_**,'了', '”', '。'], ['w', 'v', 'r', 'p', 'r', 'n', 'f', 'u', 'n', 'd', 'v', 'v', 'LOC', 'u', 'n', 'a', 'u', 'w', 'w']]
### 尝试:
1. 把`seg_model\conf`下`word.dic` 替换为 `lac_model\conf`下`word.dic`,会报错,不支持多字词;
2. 把 `lac_model\conf`下`word.dic`替换为`seg_model\conf`下`word.dic` ,结果一致。
3. 把 `rank_model\conf`下`word.dic`替换为`seg_model\conf`下`word.dic`,rank不受其影响,依赖于lac模式的分词结果。
### 能否在三种模式下使用同一个字典文件来确保分词结果一致?
实在不想要维护三个版本的字典文件,可能会造成混乱啊。
lac模式下只用单字的字典文件会不会有影响?
使用三个版本的字典文件,是有什么考虑么?
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the three-mode example using the pip-installed LAC package, then compare lac_model/conf/word.dic, seg_model/conf/word.dic, and rank_model/conf/word.dic. Check how each mode loads and uses these dictionaries. Done means the supported dictionary relationship and its effect on segmentation are established and documented, or the inconsistency is corrected with verification using the reported text.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100