change(web): dictionary-based wordbreaker should take flag to enforce boundary at string's end 🔬
Open
change
epic-dict-breaker
web/
- Dominant language
- Pascal
- Stars
- 534
- Forks
- 143
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 113
Description
https://github.com/keymanapp/keyman/blob/3e50a03ea0cf49b0421f97bf34812d2e9f7d7b29/web/src/engine/predictive-text/wordbreakers/src/main/dict/index.ts#L87-L92
In some of my recent work on epic/auto-correct, keeping in mind the needs of epic/dict-breaker, it appears we'll need to address this scenario in order to properly handle changes as the context window slides backward from backspaces and similar. It's more efficient to retokenize just part of the context, rather than the full window, and we don't want a partial token by mistake when doing related updates.
Contributor guide
Assessment
This issue has not been assessed yet.