eliminate pathological performance on StopFilter when using a Set<String> instead of CharArraySet [LUCENE-2279]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
passing a Set<Srtring> to a StopFilter instead of a CharArraySet results in a very slow filter.
this is because for each document, Analyzer.tokenStream() is called, which ends up calling the StopFilter (if used). And if a regular Set<String> is used in the StopFilter all the elements of the set are copied to a CharArraySet, as we can see in it's ctor:
public StopFilter(boolean enablePositionIncrements, TokenStream input, Set stopWords, boolean ignoreCase)
{
super(input);
if (stopWords instanceof CharArraySet) {
this.stopWords = (CharArraySet)stopWords;
} else {
this.stopWords = new CharArraySet(stopWords.size(), ignoreCase);
this.stopWords.addAll(stopWords);
}
this.enablePositionIncrements = enablePositionIncrements;
init();
}
i feel we should make the StopFilter signature specific, as in specifying CharArraySet vs Set, and there should be a JavaDoc warning on using the other variants of the StopFilter as they all result in a copy for each invocation of Analyzer.tokenStream().
---
Migrated from [LUCENE-2279](https://issues.apache.org/jira/browse/LUCENE-2279) by thushara wijeratna, updated Sep 25 2011
Contributor guide
Research direction
Start with the StopFilter constructor shown in the issue and trace how Analyzer.tokenStream() reaches it for each document. Review the other StopFilter variants mentioned, then define the API and JavaDoc changes needed to prevent repeated Set-to-CharArraySet copying; done means the pathological performance is addressed and the affected usage is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100