apache / apache/lucene

CommonGramsFilter improvements [LUCENE-2841]

Open
#3,915 9 comments 0 reactions 0 assignees View on GitHub
affects-version:3.1 affects-version:4.0-ALPHA legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Currently CommonGramsFilter expects users to remove the common words around which output token ngrams are formed, by appending a StopFilter to the analysis pipeline. This is inefficient in two ways: captureState() is called on (trailing) stopwords, and then the whole stream has to be re-examined by the following StopFilter.

The current ctor should be deprecated, and another ctor added with a boolean option controlling whether the common words should be output as unigrams.

If common words **are** configured to be output as unigrams, captureState() will still need to be called, as it is now.

If the common words are **not** configured to be output as unigrams, rather than calling captureState() for the trailing token in each output token ngram, the term text, position and offset can be maintained in the same way as they are now for the leading token: using a System.arrayCopy()'d term buffer and a few ints for positionIncrement and offsetd. The user then no longer would need to append a StopFilter to the analysis chain.

An example illustrating both possibilities should also be added.

---
Migrated from [LUCENE-2841](https://issues.apache.org/jira/browse/LUCENE-2841) by Steven Rowe (@sarowe), 2 votes, updated May 09 2016
Attachments: [commit-6402a55.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2841/commit-6402a55.patch)

Contributor guide

Open the contributing guide

Research direction

Start with CommonGramsFilter and its current constructor, then compare the analysis behavior with StopFilter as described. Add the constructor option and update token handling for both unigram settings; document both possibilities with an example and verify that the extra StopFilter is unnecessary when unigrams are disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.