其他语言分词怎么处理？
Open
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 657
- PR merge metrics
- No merged PRs in 30d
Description
类似英语等其他语言，用lac分词不太准确。语料是否可以预分词再输出，如何操作？如果不行，怎么更改分词插件。谢谢~
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points. First determine how AnyQ invokes LAC and whether pre-tokenized corpora are supported; then document the supported path or scope a tokenizer-plugin change and validate it with a representative query in another language.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100