AppFlowy-IO / AppFlowy-IO/AppFlowy
[Bug] CJK full-text search does not work
- Ngôn ngữ chính
- Dart
- Star
- 76.6k
- Fork
- 6k
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
### Bug Description
The CJK full-text search is currently not functioning as expected.
This is likely due to the Tantivy index using the default tokenizer, which splits text only by whitespace and punctuation. Because CJK languages lack word-bounding spaces, the entire text might be getting indexed as a single giant token.
### How to Reproduce
1. Create a database containing Japanese text (e.g., "今日はいい天気です").
2. Search for a substring that is not at the beginning of the text (e.g., "天気").
3. Notice that no results are returned.
### Expected Behavior
Searching for any word or phrase present in a database record should return that record, regardless of its position within the string.
### Operating System
Windows 11
### AppFlowy Version(s)
0.12.1
### Screenshots
https://github.com/user-attachments/assets/dd1a5d64-bb30-49c0-b0eb-38b93f15d89c
### Additional Context
_No response_
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.