apache / apache/lucene

Question for nori analyer behavior change

Open
#14,637 0 comments 0 reactions 0 assignees View on GitHub
type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Hello Maintainers of Nori Analyzer,

Recently, we upgraded ElasticSearch from v6.8 to v8.16 and installed Nori analyzer plugin corresponds with v8.16 ElasticSearch.

And, through some tests, we noticed that the tokenization behavior of the Nori analyzer was different.

For example, in the case of "A7B" word, which is the Nori analyzer of Elastic Search v6.8, the token is tokenized as A7B.
However, using ElasticSearch v8.16 tokenizes this word to A, 7, B.

So, I would like to ask you some questions below.

1) Reason or background of this change.
What are the benefits of this change?

2) Is there a way to configure a tokenization method that is the same or similar to the Nori analyzer of ElasticSearch V6.8?

current ES 8> installed plugin
analysis-nori-8.16.4.jar lucene-analysis-nori-9.12.0.jar

previous ES 6> installed plugin
analysis-nori-6.8.2.jar lucene-analyzers-nori-7.7.0.jar

Contributor guide

Open the contributing guide

Research direction

Reproduce the A7B tokenization difference with analysis-nori-6.8.2/lucene-analyzers-nori-7.7.0 and analysis-nori-8.16.4/lucene-analysis-nori-9.12.0. Trace the Nori analyzer behavior across those versions and document the reason for the change, its benefits, and whether configuration can restore the earlier tokenization.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.