Support empty alphabet, for simple CJK word segmentation
Open
tokenisation
- Dominant language
- C++
- Stars
- 21
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
Before
https://github.com/apertium/lttoolbox/commit/944ed2556c38f058a5118ab5e481b3412aa3e3d8 / https://github.com/apertium/lttoolbox/pull/52
it was possible to use monodix files with an empty `` in order to segment into all known analyses (presumably symbols without analyses were output as blanks). But after the change, this is no longer possible.
See https://github.com/apertium/lttoolbox/commit/944ed2556c38f058a5118ab5e481b3412aa3e3d8#commitcomment-35679780 for test cases for Chinese/Japanese/Korean.
Maybe the iswalnum test could be turned off by a flag, e.g. `lt-proc --no-implicit-alphabet` ?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.