chore(core): filter illegal characters and unpaired surrogates from app context 🌱
- Dominant language
- Pascal
- Stars
- 534
- Forks
- 143
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 113
Description
> no function for that yet but could use the new-ish macros.
_Originally posted by @srl295 in https://github.com/keymanapp/keyman/pull/10382#discussion_r1452658216_
Keyman Core does not currently filter illegal Unicode characters (particularly U+FFFF) or unpaired surrogates. These both have the potential to cause trouble in keystroke processing, particularly for LDML keyboards, so we should be filtering them as they enter Core in `km_core_state_context_set_if_needed`.
Note: if the first code unit of the UTF-16 input context in km_core_state_context_set_if_needed is the second half of a surrogate pair, we need to track forward one code unit in order to exclude it from subsequent processing, rather than patching it out.
Note test case `test_context_normalization_invalid_unicode` , needs to be updated so the invalid char is not the last character in the context, due to the change made with fix(core): skip leading trail surrogate char in km_core_state_context_set_if_needed(
Contributor guide
Assessment
This issue has not been assessed yet.