Put shared rules (and, separately, macros) into files to be included into StandardTokenizerImpl.jflex and UAX29URLEmailTokenizerImpl.jflex [LUCENE-5464]
Open
legacy-jira-priority:Minor
module:analysis
type:enhancement
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
The JFlex scanner specifications for StandardTokenizer and UAX29URLEmailTokenizer share macros and rules.
As of the JFlex 1.5 release, the `%include` directive can be used in the rules section, in addition to the macros section.
The shared macros and rules should be pulled out into files which are `%include`'d into the two scanner specifications.
---
Migrated from [LUCENE-5464](https://issues.apache.org/jira/browse/LUCENE-5464) by Steven Rowe (@sarowe)
Contributor guide
Assessment
This issue has not been assessed yet.