keymanapp / keymanapp/keyman

feat(web): bigram / n-gram support for lexical models

Open
#13,111 2 comments 0 reactions 0 assignees View on GitHub
feat web/ web/predictive-text/
Dominant language
Pascal
Stars
534
Forks
143
Avg merge
2d 10h
Merged PRs (30d)
113

Description

### Is your feature request related to a problem? Please describe.

This is coming out of https://github.com/keymanapp/lexical-models/pull/299 - in particular, its state as of commit https://github.com/keymanapp/lexical-models/pull/299/commits/fa4fafecfaccc5985cdbb99307d42f35e24f0b71.

The user has included sections that make clear a very strong desire for [bigram support](https://en.wikipedia.org/wiki/Bigram):

https://github.com/keymanapp/lexical-models/blob/fa4fafecfaccc5985cdbb99307d42f35e24f0b71/release/cathaylab.nan-latn-tw.taigi/source/wordlist.tsv?plain=1#L30-L89

(Also check out the [n-gram model Wikipedia page](https://en.wikipedia.org/wiki/Word_n-gram_language_model).)

Excerpt from the TSV file

```
góa ê 102
góa ū 102
góa bô 102
góa sī 102
góa tī 102
góa beh 102
góa kah 102
góa lâi 102
góa khì 101
góa chai 101
góa m̄-sī 101
góa kóng 101
góa ē 101
góa ái 101
góa lóng 101
góa bē 101
góa bōe 101
goán ê 102
goán ū 102
goán bô 102
goán sī 102
goán tī 102
goán beh 102
goán kah 102
goán lâi 102
goán khì 101
goán chai 101
goán m̄-sī 101
goán kóng 101
goán ē 101
goán ái 101
goán lóng 101
goán bē 101
goán bōe 101
gún ê 102
gún ū 102
gún bô 102
gún sī 102
gún tī 102
gún beh 102
gún kah 102
gún lâi 102
gún khì 101
gún chai 101
gún m̄-sī 101
gún kóng 101
gún ē 101
gún ái 101
gún lóng 101
gún bē 101
gún bōe 101
lán ê 102
lán ū 102
lán bô 102
lán sī 102
lán tī 102
lán beh 102
lán kah 102
lán lâi 102
lán khì 101
```

(There's actually similar structure with some of the model's hyphenated words worth considering as well.)

There are some pretty clear patterns suggested by that specific section we might be able to capitalize on. Either way, it's pretty clear that the implied request is the ability to have the previously-typed word tailor the current word's predictions in some manner, at least in certain cases.

### Describe the solution you'd like

Copying the excerpt above...

Excerpt from the TSV file

```
góa ê 102
góa ū 102
góa bô 102
góa sī 102
góa tī 102
góa beh 102
góa kah 102
góa lâi 102
góa khì 101
góa chai 101
góa m̄-sī 101
góa kóng 101
góa ē 101
góa ái 101
góa lóng 101
góa bē 101
góa bōe 101
goán ê 102
goán ū 102
goán bô 102
goán sī 102
goán tī 102
goán beh 102
goán kah 102
goán lâi 102
goán khì 101
goán chai 101
goán m̄-sī 101
goán kóng 101
goán ē 101
goán ái 101
goán lóng 101
goán bē 101
goán bōe 101
gún ê 102
gún ū 102
gún bô 102
gún sī 102
gún tī 102
gún beh 102
gún kah 102
gún lâi 102
gún khì 101
gún chai 101
gún m̄-sī 101
gún kóng 101
gún ē 101
gún ái 101
gún lóng 101
gún bē 101
gún bōe 101
lán ê 102
lán ū 102
lán bô 102
lán sī 102
lán tī 102
lán beh 102
lán kah 102
lán lâi 102
lán khì 101
```

This user's expectation was for wordlist entries like these to result in bigram-oriented modeling, with the pre-whitespace word auto-generating the right-hand word as a follow-up prediction.

As a personal note, aspects of this remind me of some of my own ongoing thoughts toward agglutinative model support (as with #3058.) There's a fair chance that there are common followup sets, as visible in the data excerpted above, that could be encoded in the model to facilitate such patterns.

### Describe alternatives you've considered

The user attempted to add a `[ ]` (space) entry into the models' `joinWordsAt` array in an attempt to force bi-gram support. It's simply not tailored to that, though.

### Related issues

https://github.com/keymanapp/lexical-models/pull/299

### Keyman apps

- [x] Keyman for Android
- [x] Keyman for iPhone and iPad
- [ ] Keyman for Linux
- [ ] Keyman for macOS
- [ ] Keyman for Windows
- [ ] Keyman Developer
- [x] KeymanWeb
- [ ] Other - give details at bottom of form

### Keyman version

_No response_

### Operating system

_No response_

### Device

_No response_

### Target application

_No response_

### Browser

_No response_

### Keyboard name

_No response_

### Keyboard version

_No response_

### Language name

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.