apache / apache/lucene

Should EdgeNGramTokenizer's DEFAULT_MAX_GRAM_SIZE be ONE?

Open
#13,802 2 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

From org.apache.lucene:lucene-analysis-common:9.11.1, the static variable `DEFAULT_MAX_GRAM_SIZE` of EdgeNGramTokenizer is ONE not TWO.

Logically, the maximum n-gram size must be >= minGramSize and it's not a problem but **NOT PRACTICAL**.

Since many libraries(**git code**: [Elasticsearch](https://github.com/elastic/elasticsearch/blob/main/modules/analysis-common/src/main/java/org/elasticsearch/analysis/common/CommonAnalysisPlugin.java#L511), [OpenSearch](https://github.com/opensearch-project/OpenSearch/blob/main/modules/analysis-common/src/main/java/org/opensearch/analysis/common/EdgeNGramTokenizerFactory.java#L54)) use `NGramTokenizer.DEFAULT_MAX_NGRAM_SIZE` not `EdgeNGramTokenizer`'s.
Will there be any dependency problem in Lucene project as a result of my suggestion?

See the below codes:

```java
public class EdgeNGramTokenizer extends NGramTokenizer {
public static final int DEFAULT_MAX_GRAM_SIZE = 1; /* How about changing '1' to '2'? */
public static final int DEFAULT_MIN_GRAM_SIZE = 1;

public EdgeNGramTokenizer(int minGram, int maxGram) {
super(minGram, maxGram, true);
}

public EdgeNGramTokenizer(AttributeFactory factory, int minGram, int maxGram) {
super(factory, minGram, maxGram, true);
}
}

```

Contributor guide

Open the contributing guide

Research direction

Start with EdgeNGramTokenizer and its DEFAULT_MAX_GRAM_SIZE and DEFAULT_MIN_GRAM_SIZE declarations, then compare them with NGramTokenizer.DEFAULT_MAX_NGRAM_SIZE. Review the Elasticsearch and OpenSearch usages linked in the issue and assess compatibility implications of changing the default from one to two. Done means the dependency impact and the appropriate Lucene change are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.