element-hq / element-hq/element-web
Lack of i18n support for full-text search in encrypted rooms
- Dominant language
- TypeScript
- Stars
- 13.5k
- Forks
- 2.8k
- PR merge metrics
- PR metrics pending
Description
### Steps to reproduce
1. Change the language to English
3. Click Settings -> Security and Privacy -> Message search -> Disable
4. Click Settings -> Security and Privacy -> Message search -> Enable
5. Create a new room and post a Japanese message `この部屋は全文検索用のテスト`.
6. Wait until all the rooms are indexed in Settings -> Security and Privacy -> Message search.
7. Search `部屋` in the room.
### Outcome
#### What did you expect?
The word `部屋` is found.
#### What happened instead?
No result found.
#### My thoughts
This appears to stem from the lack of i18n support for full-text search in encrypted rooms.
The full-text search is backed by Matrix-Seshat.
I looked into its source code and found that a custom tokenizer is not supported for Japanese (or many other Asian languages).
https://github.com/matrix-org/seshat/blob/main/src/config.rs#L171
Furthermore, the language is not passed from Element desktop to Matrix-seshat:
https://github.com/element-hq/element-desktop/blob/develop/src/seshat.ts#L112
If my understanding is correct, there are two problems:
* No tokenizer for Asian languages
* Correct language is not passed to Matrix-Seshat
I am using multiple languages (English and Japanese) in many rooms.
That means the language may differ from message to message (from room to room).
To support this case, either of the following options seems to be reasonable:
* For each event/message, we detect the language and use the corresponding tokenizer.
* Use an NN-based multilingual tokenizer
* Use an advanced full-search library with CJK support in JS, e.g., FlexSearch
I am a computational physicist & scientific programmer, but I am not familiar with this kind of business.
Any help or suggestions would be greatly appreciated.
### Operating system
Mac
### Application version
Element Desktop 1.11.101
### How did you install the app?
From the official website
### Homeserver
matrix.org
### Will you send logs?
No
Contributor guide
Assessment
This issue has not been assessed yet.