Tokenization status error
Open
bug
- Dominant language
- C++
- Stars
- 119
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Description
The following Modex rule does not match "Independent JPEG Group" while it should:
`@organizationKey:(t_capital_1st|t_capital|t_capital_small|\&){1-5}::ORGANIZATION:`
Instead, it matches only "JPEG Group".
This is due to the fact that the tokenzation status of "Independent" is wrongly set to
```
t_not_roman
t_alphanumeric
```
instead of
```
t_capital_1st
t_capital_1st
```
The problem comes probably from the tokenizer starting to identify "I" as part of a roman number and then wrongly backtracking.
Contributor guide
Assessment
This issue has not been assessed yet.