feat(web): improve default wordbreaker handling near the text insertion point
- Dominant language
- Pascal
- Stars
- 534
- Forks
- 143
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 113
Description
Prior related work:
- Wordbreaking:
- https://github.com/keymanapp/keyman/pull/3032
- https://github.com/keymanapp/keyman/pull/6574
- https://github.com/keymanapp/keyman/pull/7279
- Correction search batching:
- https://github.com/keymanapp/keyman/pull/14592
Relates to:
- #14709
Over time, we've added some infrastructure to work around some of the "rough edges" for word-breaking & context-tokenization within the predictive-text worker. Looking into proper modeling of alternate keystroke sequences, what we currently have does not look sufficient to model cases where context tokenization is ambiguous - where more context is needed before 100% committing to a wordbreak for text before a caret.
Consider `can'` => `can't`.
- When appearing in the middle of context, as in `he said 'yes, I can' sarcastically`, `can'` is clearly two separate tokens - `can` + a closing single-quote.
- When appearing right before the text insertion point, it's less clear. The user _could_ be typing something similar and ending a quote... or they may be about to finish typing a contraction.
----
Additionally, this causes some issues for "true correction" efforts due to the engine's design - the correction-search engine uses batching of similar tokenizations and input transforms to keep logic overhead and memory use costs down.
Examining https://unicode.org/reports/tr29/#Word_Boundaries, which is the basis of our default wordbreaker, it would appear that this scenario arises for rules that require _two_ right-hand characters (and associated properties) for application. With `can'` above, the rule that keeps `can't` as a single token can't apply yet because nothing is in the `t` position.
But... what if we could note that there's the _potential_ for a rule to apply afterward? The fact that a rule needing two right-hand characters matches everywhere but on that character - that's significant and could facilitate noting an ambiguous wordbreaker case.
Also of note is that the wordbreaker currently does not provide empty tokens for certain cases that will _always_ wordbreak. For example, there are no default wordbreaker rules that could prevent wordbreaking immediately after a `?` or a `!`. (`.` and `,` are included in MidLetter and MidNum respectively, though.) We do not currently signal an empty token following either mark when either is immediately before the text insertion point.
----
The combination of these wordbreaking effects with correction-search batching leads to an interesting scenario: suppose that we the context `he` and an incoming fat-finger distribution including the following text changes:
- standard letters: `l`, `p`
- punctuation marks: `;`, `/`
- a space: ` `
- the single quote: `'`
Results:
- If the _primary input_ - the actually-typed key - is `l` or `p`, we do not add a new word boundary.
- If the _primary input_ is one of the two punctuation marks listed or the space, we do add a new word boundary.
- If that input is instead the `'`, it's ambiguous which case is correct.
Furthermore, note the implications if we get the same distribution of keystrokes _again_ to the resulting context! If we say that the single quote should not wordbreak, in case a contraction is being typed, there are clear cases for which it _should have been_ wordbroken for certain incoming fat-fingers. The same can also be said if we assumed that it _should_ have been wordbroken - `he'l` can become `he'll`, a valid contraction!
The current batching design does not currently provide a way to adequately handle such ambiguous cases.
> But... what if we could note that there's the _potential_ for a rule to apply afterward? The fact that a rule needing two right-hand characters matches everywhere but on that character - that's significant and could facilitate noting an ambiguous wordbreaker case.
This tidbit could lead us to a solution, though - we could construct three sets for incoming fat-finger inputs, loosely speaking:
- the end of the input _does_ trigger a wordbreak
- new input does _not_ trigger a wordbreak
- new input _might_ trigger a wordbreak, but it's ambiguous.
The last set above would need to be specially evaluated for each possible follow-up input variant, unlike the other two cases. That said, it would nearly always be quite a small set - the sets of values taking up the first right-hand context slot for two-slot word-breaking rules are quite specialized. This dramatically restricts the possible permutations for such cases, preventing us from needing an O(N^2) operation for batching transition effects for all but the worst cases.
----
My current thoughts:
https://github.com/keymanapp/keyman/blob/164c1e811efa385bf04beb6620fe975df22d7175/web/src/engine/predictive-text/wordbreakers/src/main/default/index.ts#L89-L107
https://github.com/keymanapp/keyman/blob/164c1e811efa385bf04beb6620fe975df22d7175/web/src/test/auto/headless/engine/predictive-text/wordbreakers/default-word-breaker.tests.ts#L203-L216
A better design would be for the `WordbreakerRule` type would be to describe the context patterns that the rule may fit. Where the current `match` method specifies them internally within method calls, the array parameters _of_ those method calls holds the data we need to check for the ambiguous cases mentioned above... as well as for cases like `?` and `!` for which we can guarantee an extra wordbreak between final character and text insertion point.
Contributor guide
Assessment
This issue has not been assessed yet.