element-hq / element-hq/element-meta

i18n support for full-text search in encrypted rooms

Open
#2,883 7 comments 1 reaction 0 assignees View on GitHub
T-Enhancement
Dominant language
No language data
Stars
112
Forks
25
Avg merge
6h 6m
Merged PRs (30d)
4

Description

### Your use case

#### What would you like to do?
I am in many rooms.
The primary language differs from room to room.
In some cases, a message may contain sentences in different languages.

I want to search for a word in a non-Western language in such an encrypted room.

#### Why would you like to do it?
This situation is common when using Element to chat with friends from different countries.
But, in the current implementation, searching for a non-Western word is very unreliable and misses most of the hits.
See the discussion in https://github.com/element-hq/element-web/issues/32038#issuecomment-2939060015.

#### How would you like to achieve it?
The current implementation in Seshat is based on a morphological analysis of a global language-specific tokenizer.
The tokenizer (and the language) is chosen when we initiate the search database.

https://github.com/element-hq/element-web/issues/32038#issuecomment-2939060015

As discussed in https://github.com/matrix-org/seshat/issues/7, it isn't easy to support multilingual situations.

I'd like to achieve a reliable (exact-match) search in multilingual situations, for example, by switching to an N-gram-based algorithm,
where we would not need language-specific tokenizers and automatic language detection for each room or each message.

I am a computational scientist & computational programmer, and need to accept that I am not an expert in search engines.
Any suggestions will be welcomed.

### Have you considered any alternatives?

Yes, but I did not see any alternatives. A search is a basic functionality of a chat system.

### Additional context

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the Seshat issue #7 and the linked Element issue discussion, focusing on the limits of the current language-specific tokenizer in multilingual encrypted rooms. Define and validate a search approach that reliably finds exact matches across languages and mixed-language messages; the issue provides no implementation files or tests to run.

Written by the indexing model from the issue text.

Assessment

Domain
internationalization, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.