element-hq / element-hq/element-meta
i18n support for full-text search in encrypted rooms
- Dominant language
- No language data
- Stars
- 112
- Forks
- 25
- Avg merge
- 6h 6m
- Merged PRs (30d)
- 4
Description
### Your use case
#### What would you like to do?
I am in many rooms.
The primary language differs from room to room.
In some cases, a message may contain sentences in different languages.
I want to search for a word in a non-Western language in such an encrypted room.
#### Why would you like to do it?
This situation is common when using Element to chat with friends from different countries.
But, in the current implementation, searching for a non-Western word is very unreliable and misses most of the hits.
See the discussion in https://github.com/element-hq/element-web/issues/32038#issuecomment-2939060015.
#### How would you like to achieve it?
The current implementation in Seshat is based on a morphological analysis of a global language-specific tokenizer.
The tokenizer (and the language) is chosen when we initiate the search database.
https://github.com/element-hq/element-web/issues/32038#issuecomment-2939060015
As discussed in https://github.com/matrix-org/seshat/issues/7, it isn't easy to support multilingual situations.
I'd like to achieve a reliable (exact-match) search in multilingual situations, for example, by switching to an N-gram-based algorithm,
where we would not need language-specific tokenizers and automatic language detection for each room or each message.
I am a computational scientist & computational programmer, and need to accept that I am not an expert in search engines.
Any suggestions will be welcomed.
### Have you considered any alternatives?
Yes, but I did not see any alternatives. A search is a basic functionality of a chat system.
### Additional context
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the Seshat issue #7 and the linked Element issue discussion, focusing on the limits of the current language-specific tokenizer in multilingual encrypted rooms. Define and validate a search approach that reliably finds exact matches across languages and mixed-language messages; the issue provides no implementation files or tests to run.
Written by the indexing model from the issue text.
Assessment
- Domain
- internationalization, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100