keymanapp / keymanapp/keyman

feat(web): improve default wordbreaker handling near the text insertion point

Open
#14,763 0 comments 0 reactions 1 assignee Claimed by @jahorton View on GitHub
feat web/
Dominant language
Pascal
Stars
534
Forks
143
Avg merge
2d 10h
Merged PRs (30d)
113

Description

Prior related work:
- Wordbreaking:
- https://github.com/keymanapp/keyman/pull/3032
- https://github.com/keymanapp/keyman/pull/6574
- https://github.com/keymanapp/keyman/pull/7279
- Correction search batching:
- https://github.com/keymanapp/keyman/pull/14592

Relates to:
- #14709

Over time, we've added some infrastructure to work around some of the "rough edges" for word-breaking & context-tokenization within the predictive-text worker. Looking into proper modeling of alternate keystroke sequences, what we currently have does not look sufficient to model cases where context tokenization is ambiguous - where more context is needed before 100% committing to a wordbreak for text before a caret.

Consider `can'` => `can't`.
- When appearing in the middle of context, as in `he said 'yes, I can' sarcastically`, `can'` is clearly two separate tokens - `can` + a closing single-quote.
- When appearing right before the text insertion point, it's less clear. The user _could_ be typing something similar and ending a quote... or they may be about to finish typing a contraction.

----

Additionally, this causes some issues for "true correction" efforts due to the engine's design - the correction-search engine uses batching of similar tokenizations and input transforms to keep logic overhead and memory use costs down.

Examining https://unicode.org/reports/tr29/#Word_Boundaries, which is the basis of our default wordbreaker, it would appear that this scenario arises for rules that require _two_ right-hand characters (and associated properties) for application. With `can'` above, the rule that keeps `can't` as a single token can't apply yet because nothing is in the `t` position.

But... what if we could note that there's the _potential_ for a rule to apply afterward? The fact that a rule needing two right-hand characters matches everywhere but on that character - that's significant and could facilitate noting an ambiguous wordbreaker case.

Also of note is that the wordbreaker currently does not provide empty tokens for certain cases that will _always_ wordbreak. For example, there are no default wordbreaker rules that could prevent wordbreaking immediately after a `?` or a `!`. (`.` and `,` are included in MidLetter and MidNum respectively, though.) We do not currently signal an empty token following either mark when either is immediately before the text insertion point.

----

The combination of these wordbreaking effects with correction-search batching leads to an interesting scenario: suppose that we the context `he` and an incoming fat-finger distribution including the following text changes:
- standard letters: `l`, `p`
- punctuation marks: `;`, `/`
- a space: ` `
- the single quote: `'`

Results:
- If the _primary input_ - the actually-typed key - is `l` or `p`, we do not add a new word boundary.
- If the _primary input_ is one of the two punctuation marks listed or the space, we do add a new word boundary.
- If that input is instead the `'`, it's ambiguous which case is correct.

Furthermore, note the implications if we get the same distribution of keystrokes _again_ to the resulting context! If we say that the single quote should not wordbreak, in case a contraction is being typed, there are clear cases for which it _should have been_ wordbroken for certain incoming fat-fingers. The same can also be said if we assumed that it _should_ have been wordbroken - `he'l` can become `he'll`, a valid contraction!

The current batching design does not currently provide a way to adequately handle such ambiguous cases.

> But... what if we could note that there's the _potential_ for a rule to apply afterward? The fact that a rule needing two right-hand characters matches everywhere but on that character - that's significant and could facilitate noting an ambiguous wordbreaker case.

This tidbit could lead us to a solution, though - we could construct three sets for incoming fat-finger inputs, loosely speaking:
- the end of the input _does_ trigger a wordbreak
- new input does _not_ trigger a wordbreak
- new input _might_ trigger a wordbreak, but it's ambiguous.

The last set above would need to be specially evaluated for each possible follow-up input variant, unlike the other two cases. That said, it would nearly always be quite a small set - the sets of values taking up the first right-hand context slot for two-slot word-breaking rules are quite specialized. This dramatically restricts the possible permutations for such cases, preventing us from needing an O(N^2) operation for batching transition effects for all but the worst cases.

----

My current thoughts:

https://github.com/keymanapp/keyman/blob/164c1e811efa385bf04beb6620fe975df22d7175/web/src/engine/predictive-text/wordbreakers/src/main/default/index.ts#L89-L107

https://github.com/keymanapp/keyman/blob/164c1e811efa385bf04beb6620fe975df22d7175/web/src/test/auto/headless/engine/predictive-text/wordbreakers/default-word-breaker.tests.ts#L203-L216

A better design would be for the `WordbreakerRule` type would be to describe the context patterns that the rule may fit. Where the current `match` method specifies them internally within method calls, the array parameters _of_ those method calls holds the data we need to check for the ambiguous cases mentioned above... as well as for cases like `?` and `!` for which we can guarantee an extra wordbreak between final character and text insertion point.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.