baidu / baidu/lac

'seg'和'lac'分词结果不一致

Open
#188 1 comment 4 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
4k
Forks
592
PR merge metrics
No merged PRs in 30d

Description

### 问题:
pip默认安装,发现`lac_model`,`seg_model`,`rank_model`下`word.dic`字典文件不同,导致分词结果不一致。

`lac_model\conf`下`word.dic`,746KB,58223行,包含多字词语;
`rank_model\conf`下`word.dic`,746KB,58223行,包含多字词语,和`lac_model`相同;
`seg_model\conf`下`word.dic`,71KB,8223行,只包含单字。

```
text = "“没有什么比这场疫情下的生与死更能体现美国的肤色差异了”。"

lac_seg = LAC(mode='seg')
seg_result = lac_seg.run(text)

lac_lac = LAC(mode='lac')
lac_result = lac_lac.run(text)

lac_rank = LAC(mode='rank')
rank_result = lac_rank.run(text)

```
### **结果:**
seg_result = ['“', '没有', '什么', '比', '这', '场', '疫情', '下', '的', '生与死', '更', '能', '体现', '美国', '的', **'肤色'**, **'差异'**, '了', '”', '。']

lac_result = [['“', '没有', '什么', '比', '这场', '疫情', '下', '的', '生与死', '更', '能', '体现', '美国', '的', **_'肤'_**, **_'色差异'_**,'了', '”', '。'], ['w', 'v', 'r', 'p', 'r', 'n', 'f', 'u', 'n', 'd', 'v', 'v', 'LOC', 'u', 'n', 'a', 'u', 'w', 'w']]

### 尝试:
1. 把`seg_model\conf`下`word.dic` 替换为 `lac_model\conf`下`word.dic`,会报错,不支持多字词;
2. 把 `lac_model\conf`下`word.dic`替换为`seg_model\conf`下`word.dic` ,结果一致。
3. 把 `rank_model\conf`下`word.dic`替换为`seg_model\conf`下`word.dic`,rank不受其影响,依赖于lac模式的分词结果。

### 能否在三种模式下使用同一个字典文件来确保分词结果一致?
实在不想要维护三个版本的字典文件,可能会造成混乱啊。
lac模式下只用单字的字典文件会不会有影响?
使用三个版本的字典文件,是有什么考虑么?

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the three-mode example using the pip-installed LAC package, then compare lac_model/conf/word.dic, seg_model/conf/word.dic, and rank_model/conf/word.dic. Check how each mode loads and uses these dictionaries. Done means the supported dictionary relationship and its effect on segmentation are established and documented, or the inconsistency is corrected with verification using the reported text.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.