Should EdgeNGramTokenizer's DEFAULT_MAX_GRAM_SIZE be ONE?
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
From org.apache.lucene:lucene-analysis-common:9.11.1, the static variable `DEFAULT_MAX_GRAM_SIZE` of EdgeNGramTokenizer is ONE not TWO.
Logically, the maximum n-gram size must be >= minGramSize and it's not a problem but **NOT PRACTICAL**.
Since many libraries(**git code**: [Elasticsearch](https://github.com/elastic/elasticsearch/blob/main/modules/analysis-common/src/main/java/org/elasticsearch/analysis/common/CommonAnalysisPlugin.java#L511), [OpenSearch](https://github.com/opensearch-project/OpenSearch/blob/main/modules/analysis-common/src/main/java/org/opensearch/analysis/common/EdgeNGramTokenizerFactory.java#L54)) use `NGramTokenizer.DEFAULT_MAX_NGRAM_SIZE` not `EdgeNGramTokenizer`'s.
Will there be any dependency problem in Lucene project as a result of my suggestion?
See the below codes:
```java
public class EdgeNGramTokenizer extends NGramTokenizer {
public static final int DEFAULT_MAX_GRAM_SIZE = 1; /* How about changing '1' to '2'? */
public static final int DEFAULT_MIN_GRAM_SIZE = 1;
public EdgeNGramTokenizer(int minGram, int maxGram) {
super(minGram, maxGram, true);
}
public EdgeNGramTokenizer(AttributeFactory factory, int minGram, int maxGram) {
super(factory, minGram, maxGram, true);
}
}
```
Contributor guide
Research direction
Start with EdgeNGramTokenizer and its DEFAULT_MAX_GRAM_SIZE and DEFAULT_MIN_GRAM_SIZE declarations, then compare them with NGramTokenizer.DEFAULT_MAX_NGRAM_SIZE. Review the Elasticsearch and OpenSearch usages linked in the issue and assess compatibility implications of changing the default from one to two. Done means the dependency impact and the appropriate Lucene change are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100