Remove [Edge]NgramTokenizer min/max defaults consistent with Filter [LUCENE-8348]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
---
Migrated from [LUCENE-8348](https://issues.apache.org/jira/browse/LUCENE-8348) by Robert Muir (@rmuir)
Environment:
```
#9009 fixed a good deal of trappiness here for the tokenfilters, there aren't ridiculous default min/max values such as 1,2.
Also javadocs are enhanced to present a nice warning about using large ranges: it seems to surprise people that min=small, max=huge eats up a ton of resources, but its really like creating (huge-small) separate n-gram indexes, so of course its expensive.
Finally it keeps it easy to do the typical, more efficient fixed ngram case, vs forcing someone to do min=X,max=X range which is unintuitive.
We should improve the tokenizers in the same way.
```
Contributor guide
Research direction
No file or test is named. Start with EdgeNgramTokenizer and compare its min/max defaults and Javadocs with the token filters discussed in LUCENE-8348, including the fixed n-gram case and warning about large ranges. Done means the tokenizer behavior and documentation follow that intended consistency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100