0xnyn / 0xnyn/blaze

No CJK support?

未关闭
#4 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Go
星标
540
派生
15
PR 合并指标
30 天内没有已合并 PR

描述

Just tested with Chinese, blaze currently does not support it. Any plan on supporting custom tokenizer (e.g. Unicode code point tokenizer)? Thanks.

```go
package main

import (
"fmt"

"github.com/wizenheimer/blaze"
)

func main() {
// Create a new inverted index
idx := blaze.NewInvertedIndex()

// Index some documents
idx.Index(1, "你好,世界")

// Search with BM25 ranking
matches := idx.RankBM25("好", 10)

// Print results
for _, match := range matches {
fmt.Printf("Document %d (score: %.2f)\n", match.DocID, match.Score)
}
}
```

贡献指南

这个仓库没有索引到贡献指南

调研方向

The issue demonstrates that Chinese text is not tokenized correctly. Examine the tokenizer in the blaze codebase, likely in a file like tokenizer.go. Research Unicode segmentation and how to integrate a custom tokenizer. A solution may involve modifying the tokenizer interface or adding a new tokenizer implementation. Testing will require verifying that Chinese characters are properly indexed and searched.

由索引模型根据 Issue 内容生成。

评估

技术栈
go
领域
search
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
描述清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。