AppFlowy-IO / AppFlowy-IO/AppFlowy

[Bug] CJK full-text search does not work

未关闭
#8,767 1 条评论 1 个 reaction 已指派 1 人 已指派给 @appflowy 在 GitHub 查看
主要语言
Dart
星标
76.6k
派生
6k
PR 合并指标
30 天内没有已合并 PR

描述

### Bug Description

The CJK full-text search is currently not functioning as expected.
This is likely due to the Tantivy index using the default tokenizer, which splits text only by whitespace and punctuation. Because CJK languages lack word-bounding spaces, the entire text might be getting indexed as a single giant token.

### How to Reproduce

1. Create a database containing Japanese text (e.g., "今日はいい天気です").
2. Search for a substring that is not at the beginning of the text (e.g., "天気").
3. Notice that no results are returned.

### Expected Behavior

Searching for any word or phrase present in a database record should return that record, regardless of its position within the string.

### Operating System

Windows 11

### AppFlowy Version(s)

0.12.1

### Screenshots

https://github.com/user-attachments/assets/dd1a5d64-bb30-49c0-b0eb-38b93f15d89c

### Additional Context

_No response_

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。