keymanapp / keymanapp/keyman

feat(core): Normalisation in Keyman Core

Open
#3,306 2 comments 0 reactions 0 assignees View on GitHub
core/ feat m:normalization
Dominant language
Pascal
Stars
534
Forks
143
Avg merge
2d 10h
Merged PRs (30d)
113

Description

# The concept

We'd love for NFC/NFD to be handled transparently within Keyman Core on all platforms.

# Background

This has been an ongoing issue. In this example, We are looking at Galaxie Greek Mnemonic interactions with FLEx, on Windows and Linux. This relates to keymanapp/keyboards#1264.

@AndreasJoswig wrote:
> But when I want to write stacked diacritics, such as ἄ or ἆ, I get very different results in FLEx:
>
> In the Windows 10 Version of FLEx (9.0.6) with Keyman 13.0, the sequence a-j-v gives the following result:
>
> ![image](https://user-images.githubusercontent.com/4498365/86305751-cf09f600-bc55-11ea-9902-753a13c03b70.png)
>
> The second diacritic does not appear over the letter, but in a new slot to the right of it. This also happens to capital vowels, where the stacked diacritic should be entirely to the left of the letter (Ἄ), but the second component goes to the right (Ἀ´).
>
> The same sequence a-j-v in Wasta Linux with FLEx 9.0.8 gives me this:
>
> ![image](https://user-images.githubusercontent.com/4498365/86305758-d3ceaa00-bc55-11ea-9185-31da0b5a3740.png)
>
> Now here the two diacritics are stacked correctly, but at the price of a second vowel character inserted before the whole sequence.

The issue in both Windows and Linux has the same root cause, albeit with different side-effects. What is happening is that the Keyman keyboard emits composed characters such as U+1F01 ἁ. However, behind the scenes FLEx turns this into a decomposed U+03B1 U+0485 α+҅.

Now, on Windows, Keyman sees the decomposed characters from the text store but the Greek Mnemonic keyboard does not have rules to support the decomposed forms, so it generates the isolated diacritic instead.

On Linux, however, Keyman does not have access to the text store, so it doesn't even know that FLEx has decomposed the string internally. So it emits a single backspace to delete what it believes is in the text store (a precomposed U+1F01), resulting in the output you see.

For Windows, we can fix this most rapidly by adding rules to the Galaxie Greek Mnemonic layout to support decomposed context matches. This would probably be a day's work with testing. (I am not sure if any of the other Greek keyboards have NFD context rules in place or not). (Feel free to open an issue on the keyboards repository).

A far better solution (which is in the roadmap, at least in theory) is adding support for composition and decomposition internally to Keyman Engine -- this is a much bigger job. Keyman would always emit a consistent normalisation form, but would transparently handle either form in the context for keyboard rules. The keyboard compiler would be updated to ensure that keyboards are internally consistent with NFC or NFD (probably configurable on the part of the keyboard author, for a number of legacy reasons). On Windows, because Keyman interacts closely with the FLEx text store, this would solve these normalisation issues -- and improve the situation in many other apps as well.

However, on Linux, we don't really have a great solution on the Keyman side at this time -- Keyman does not have access to the text store for FLEx so it just has no way of knowing that things have changed. The long term best solution would be to introduce a communication layer between Keyman and FLEx (hopefully standardised in some way...) so that Keyman can see the text store and manipulate it more directly, rather than emitting backspaces and then new characters, as it does now. That, together with the fix described above, would be enough to solve the issue here as well.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.