keymanapp / keymanapp/keyman

feat(common/models): allow for customizations to default word breaker

Open
#3,347 3 comments 0 reactions 1 assignee Claimed by @jahorton View on GitHub
developer/ developer/compilers/ feat web/predictive-text/
Dominant language
Pascal
Stars
534
Forks
143
Avg merge
2d 10h
Merged PRs (30d)
113

Description

**Is your feature request related to a problem? Please describe.**

The default wordbreaker uses the Unicode default word boundary, which explicitly states

> It is not possible to provide a uniform set of rules that resolves all issues across languages or that handles all ambiguous situations within a given language. The goal for the specification presented in this annex is to provide a workable default; **tailored implementations can be more sophisticated**. [[Source]](https://unicode.org/reports/tr29/#Word_Boundaries#:~:text=It%20is%20not%20possible%20to%20provide%20a%20uniform%20set%20of%20rules%20that%20resolves%20all%20issues%20across%20languages%20or%20that%20handles%20all%20ambiguous%20situations%20within%20a%20given%20language.%20The%20goal%20for%20the%20specification%20presented%20in%20this%20annex%20is%20to%20provide%20a%20workable%20default;%20tailored%20implementations%20can%20be%20more%20sophisticated.)

Additionally, it states:

> For Thai, Lao, Khmer, Myanmar, and other scripts that do not typically use spaces between words, **a good implementation should not depend on the default word boundary specification**. [[Source]](https://unicode.org/reports/tr29/#Word_Boundaries#:~:text=For%20Thai%2C%20Lao) (emphasis mine)

**Describe the solution you'd like**

The default word boundary rules are labelled and numbered. Make it easy to specify new rules to the specification, as in this `.model.ts` file:

```typescript
const source: LexicalModelSource = {
format: 'trie-1.0',
wordBreaker: {
use: 'default',
tailorings: {
definitions: {
Lao: /\u0e81-\u0edf/,
},
rules: {
'5a': ['Lao', NO_BOUNDARY, 'Lao'],
}
}
},
sources: ['wordlist.tsv'],
};
export default source;
```

The rules would amend the existing rules in the [default word boundary rule table](https://unicode.org/reports/tr29/#Word_Boundary_Rules).

I'm not sure what the rule syntax will end up looking like, but it would be based off the pre-existing word boundary specification. It would use these types:

```typescript
// See: https://unicode.org/reports/tr29/#Table_Boundary_Symbols
enum Operation {
BOUNDARY = '÷', // allow break here
NO_BOUNDARY = '×', // do not allow break here
};

type Definition = string;

type WordBoundaryRule = (Definition | Operation)[];
```

With the runtime constraint that `WordBoundaryRule` must contain one or more definitions, followed by a single operation, then followed by one or more definitions.

**Describe alternatives you've considered**

See: https://github.com/keymanapp/keyman/issues/3318#issuecomment-657312510

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.