apache / apache/lucene

eliminate pathological performance on StopFilter when using a Set<String> instead of CharArraySet [LUCENE-2279]

Open
#3,355 14 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

passing a Set<Srtring> to a StopFilter instead of a CharArraySet results in a very slow filter.
this is because for each document, Analyzer.tokenStream() is called, which ends up calling the StopFilter (if used). And if a regular Set<String> is used in the StopFilter all the elements of the set are copied to a CharArraySet, as we can see in it's ctor:

public StopFilter(boolean enablePositionIncrements, TokenStream input, Set stopWords, boolean ignoreCase)
{
super(input);
if (stopWords instanceof CharArraySet) {
this.stopWords = (CharArraySet)stopWords;
} else {
this.stopWords = new CharArraySet(stopWords.size(), ignoreCase);
this.stopWords.addAll(stopWords);
}
this.enablePositionIncrements = enablePositionIncrements;
init();
}

i feel we should make the StopFilter signature specific, as in specifying CharArraySet vs Set, and there should be a JavaDoc warning on using the other variants of the StopFilter as they all result in a copy for each invocation of Analyzer.tokenStream().

---
Migrated from [LUCENE-2279](https://issues.apache.org/jira/browse/LUCENE-2279) by thushara wijeratna, updated Sep 25 2011

Contributor guide

Open the contributing guide

Research direction

Start with the StopFilter constructor shown in the issue and trace how Analyzer.tokenStream() reaches it for each document. Review the other StopFilter variants mentioned, then define the API and JavaDoc changes needed to prevent repeated Set-to-CharArraySet copying; done means the pathological performance is addressed and the affected usage is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.