keymanapp / keymanapp/keyman

change(web): do not implicitly specify Trie node + entry component lengths 💾

Open
#13,696 5 comments 0 reactions 1 assignee Claimed by @jahorton View on GitHub
change epic-model-encoding web/ web/predictive-text/
Dominant language
Pascal
Stars
534
Forks
143
Avg merge
2d 10h
Merged PRs (30d)
113

Description

While we do get fantastic Trie compression by trimming redundancy and leaving certain encoding-window lengths implicit, it's worth noting the assumptions that _make_ it implicit. Saying that the encoding for lengths is totally redundant asserts that there will _never_ be any data comprising the encoded object after that variable-length section.

Should we ever wish to support versions of the Trie with additional data on such objects... where can we safely encode that data? There's no clear answer to where the range for new data may begin _unless_ we make that formerly-implicit value _explicit_ instead. If it's explicit, we then have a very clear 'start' point for the extension's data range. Therefore, it's better to just be explicit.

Refer to https://github.com/keymanapp/keyman/pull/12293#issuecomment-2791372741 - in particular, in relation to support for advanced model types, such as those leveraging Hunspell spell-checking specifications or similar. The various prefix/suffix support classes would be _very_ useful to have encoded on Trie nodes and entries - making these something we'd want to append after the standard bit-fields for these Trie objects, invalidating the assumption highlighted in the top paragraph.

----

The one twist I _could_ see on this is specifying a top-level version flag: one specific Trie version could leave these values implicit, with all others turning it explicit. But then we need a top-level version flag we're not currently encoding.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.