element-hq / element-hq/synapse

Figure out what we're doing with ICU tokenisation and locales

Open
#15,124 0 comments 0 reactions 0 assignees View on GitHub
A-I18n A-Message-Search A-Search A-User-Directory T-Other
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#15124](https://github.com/matrix-org/synapse/issues/15124).

---

The ICU tokenisation rules seem to vary on different platforms.

Is it the ICU version? The locale? (How does ICU even get a default locale? I had a quick spelunk in the source code and couldn't find it!)

We need to figure out:

- what we actually want from the ICU library
- how we get that
- how we get consistent results.

It feels like we want a 'universal' locale independent of the host's settings, so that Synapse works well with all languages. (This may be a pie in the sky goal!)
What's the best we can do?

This issue was originally dug up in #15079, but e.g. Patrick's machine generates another tokenisation yet again. I'm not satisfied with the current solution..

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.