No CJK support?
- 主要语言
- Go
- 星标
- 540
- 派生
- 15
- PR 合并指标
- 30 天内没有已合并 PR
描述
Just tested with Chinese, blaze currently does not support it. Any plan on supporting custom tokenizer (e.g. Unicode code point tokenizer)? Thanks.
```go
package main
import (
"fmt"
"github.com/wizenheimer/blaze"
)
func main() {
// Create a new inverted index
idx := blaze.NewInvertedIndex()
// Index some documents
idx.Index(1, "你好,世界")
// Search with BM25 ranking
matches := idx.RankBM25("好", 10)
// Print results
for _, match := range matches {
fmt.Printf("Document %d (score: %.2f)\n", match.DocID, match.Score)
}
}
```
贡献指南
这个仓库没有索引到贡献指南
调研方向
The issue demonstrates that Chinese text is not tokenized correctly. Examine the tokenizer in the blaze codebase, likely in a file like tokenizer.go. Research Unicode segmentation and how to integrate a custom tokenizer. A solution may involve modifying the tokenizer interface or adding a new tokenizer implementation. Testing will require verifying that Chinese characters are properly indexed and searched.
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- go
- 领域
- search
- Issue 类型
- 功能
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 描述清楚
- 新手友好度
- 35/100