Special tag for all unrecognized symbols
Open
enhancement
tokenisation
- Dominant language
- C++
- Stars
- 21
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
Hello!
There are so many non-whitespace symbols that are not recognized by Apertium's tagger and not marked in any way. For example, apertium-tat does not recognize the following symbols:
_ @ % ~ |
and thousands others.
Is it possible to use some special tag (^_/_<unknown> or <sym>$) for such cases?
Without tagging it is difficult to process Apertium's output. Streamparser also leaves such cases in "blank" variable or skips them.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.