Tokenizer eats hyphen between words
Open
question
- Dominant language
- C++
- Stars
- 119
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Description
Line "June-July 2000" is tokenized into "June", "July", "2000". Hyphen disappeared.
Probably not a bug. But the problem happens in case you need to distinguish "June July" from "June-July" on rule level.
Contributor guide
Assessment
This issue has not been assessed yet.