Question for nori analyer behavior change
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Hello Maintainers of Nori Analyzer,
Recently, we upgraded ElasticSearch from v6.8 to v8.16 and installed Nori analyzer plugin corresponds with v8.16 ElasticSearch.
And, through some tests, we noticed that the tokenization behavior of the Nori analyzer was different.
For example, in the case of "A7B" word, which is the Nori analyzer of Elastic Search v6.8, the token is tokenized as A7B.
However, using ElasticSearch v8.16 tokenizes this word to A, 7, B.
So, I would like to ask you some questions below.
1) Reason or background of this change.
What are the benefits of this change?
2) Is there a way to configure a tokenization method that is the same or similar to the Nori analyzer of ElasticSearch V6.8?
current ES 8> installed plugin
analysis-nori-8.16.4.jar lucene-analysis-nori-9.12.0.jar
previous ES 6> installed plugin
analysis-nori-6.8.2.jar lucene-analyzers-nori-7.7.0.jar
Contributor guide
Research direction
Reproduce the A7B tokenization difference with analysis-nori-6.8.2/lucene-analyzers-nori-7.7.0 and analysis-nori-8.16.4/lucene-analysis-nori-9.12.0. Trace the Nori analyzer behavior across those versions and document the reason for the change, its benefits, and whether configuration can restore the earlier tokenization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100