aymara / aymara/lima

Tokenization status error

Open
#89 0 comments 0 reactions 2 assignees Claimed by @romaricb View on GitHub
bug
Dominant language
C++
Stars
119
Forks
20
PR merge metrics
No merged PRs in 30d

Description

The following Modex rule does not match "Independent JPEG Group" while it should:
`@organizationKey:(t_capital_1st|t_capital|t_capital_small|\&){1-5}::ORGANIZATION:`

Instead, it matches only "JPEG Group".

This is due to the fact that the tokenzation status of "Independent" is wrongly set to
```



t_not_roman

t_alphanumeric

```

instead of
```


t_capital_1st


t_capital_1st

```

The problem comes probably from the tokenizer starting to identify "I" as part of a roman number and then wrongly backtracking.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.