keymanapp / keymanapp/keyman

chore(core): filter illegal characters and unpaired surrogates from app context 🌱

Open
#10,392 3 comments 0 reactions 1 assignee Claimed by @ermshiperete View on GitHub
chore core/
Dominant language
Pascal
Stars
534
Forks
143
Avg merge
2d 10h
Merged PRs (30d)
113

Description

> no function for that yet but could use the new-ish macros.
_Originally posted by @srl295 in https://github.com/keymanapp/keyman/pull/10382#discussion_r1452658216_

Keyman Core does not currently filter illegal Unicode characters (particularly U+FFFF) or unpaired surrogates. These both have the potential to cause trouble in keystroke processing, particularly for LDML keyboards, so we should be filtering them as they enter Core in `km_core_state_context_set_if_needed`.

Note: if the first code unit of the UTF-16 input context in km_core_state_context_set_if_needed is the second half of a surrogate pair, we need to track forward one code unit in order to exclude it from subsequent processing, rather than patching it out.

Note test case `test_context_normalization_invalid_unicode` , needs to be updated so the invalid char is not the last character in the context, due to the change made with fix(core): skip leading trail surrogate char in km_core_state_context_set_if_needed(

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.