keymanapp / keymanapp/keyman

chore(web): global input normalization in lexical models

Open
#9,598 4 comments 0 reactions 0 assignees View on GitHub
chore m:normalization web/ web/predictive-text/
Dominant language
Pascal
Stars
534
Forks
143
Avg merge
2d 10h
Merged PRs (30d)
113

Description

> 5. Then, once we have (4) done, the model engine should inherit the normalization form of the associated keyboard -- so if the keyboard emits NFC, the model engine should normalize(nfc) its outputs. And the converse for NFD. The internals don't matter so much, but we use NFC throughout, so let's stick with that. Inputs to the model should be normalized to NFC (they probably already are?)

The lexical model compiler already does the normalization to NFC at build time, so I think we can declare this issue done. Remaining work is in referenced issues.

_Originally posted by @mcdurdin in https://github.com/keymanapp/keyman/issues/2880#issuecomment-1726770721_

If using a Trie-based wordlist, part of the model functionality converts both sides to NFD (with the default search-term keyer) when doing a word lookup. It's consistent and has been working well, so no worries here.

For custom models, and on the global level... we technically don't yet enforce a normalization pattern.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.