apache / apache/lucene

Remove [Edge]NgramTokenizer min/max defaults consistent with Filter [LUCENE-8348]

Open
#9,395 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:analysis type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

---
Migrated from [LUCENE-8348](https://issues.apache.org/jira/browse/LUCENE-8348) by Robert Muir (@rmuir)
Environment:
```
#9009 fixed a good deal of trappiness here for the tokenfilters, there aren't ridiculous default min/max values such as 1,2.

Also javadocs are enhanced to present a nice warning about using large ranges: it seems to surprise people that min=small, max=huge eats up a ton of resources, but its really like creating (huge-small) separate n-gram indexes, so of course its expensive.

Finally it keeps it easy to do the typical, more efficient fixed ngram case, vs forcing someone to do min=X,max=X range which is unintuitive.

We should improve the tokenizers in the same way.
```

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start with EdgeNgramTokenizer and compare its min/max defaults and Javadocs with the token filters discussed in LUCENE-8348, including the fixed n-gram case and warning about large ranges. Done means the tokenizer behavior and documentation follow that intended consistency.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.