element-hq / element-hq/synapse
Figure out what we're doing with ICU tokenisation and locales
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#15124](https://github.com/matrix-org/synapse/issues/15124).
---
The ICU tokenisation rules seem to vary on different platforms.
Is it the ICU version? The locale? (How does ICU even get a default locale? I had a quick spelunk in the source code and couldn't find it!)
We need to figure out:
- what we actually want from the ICU library
- how we get that
- how we get consistent results.
It feels like we want a 'universal' locale independent of the host's settings, so that Synapse works well with all languages. (This may be a pie in the sky goal!)
What's the best we can do?
This issue was originally dug up in #15079, but e.g. Patrick's machine generates another tokenisation yet again. I'm not satisfied with the current solution..
Contributor guide
Assessment
This issue has not been assessed yet.