EducationalTestingService / EducationalTestingService/rstfinder
tokenization issues for non-ascii texts
Open
- Dominant language
- Python
- Stars
- 128
- Forks
- 24
- PR merge metrics
- No merged PRs in 30d
Description
The NLTK tokenizer used in the code doesn't handle fancy quotation marks very well. They just end up attached to words rather than being separate tokens.
We should probably either preprocess the input that is passed to the tokenizer, find another tokenizer, or fix the current one.
There may be some issues related to other types of symbols as well.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.