apache / apache/lucene

Handle stop words that appear at articulation points [LUCENE-8717]

Open
#9,763 5 comments 0 reactions 1 assignee Claimed by @romseygeek View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Our set of TokenFilters currently cannot handle the case where a multi-term synonym starts with a stopword.  This means that given a synonym file containing the mapping "the walking dead => twd" and a standard english stopword filter, QueryBuilder will produce incorrect queries.

The tricky part here is that our standard way of dealing with stopwords, which is to just remove them entirely from the token stream and use a larger position increment on subsequent tokens, doesn't work when the removed token also has a position length greater than 1.  There are various tricks you can do to increment position length on the previous token, but this doesn't work if the stopword is the first token in the token stream, or if there are multiple stopwords in the side path.

Instead, I'd like to propose adding a new TermDeletedAttribute, which we only use on tokens that should be removed from the stream but which hold necessary information about the structure of the token graph.  These tokens can then be removed by GraphTokenStreamFiniteStrings at query time, and by FlattenGraphFilter at index time.

---
Migrated from [LUCENE-8717](https://issues.apache.org/jira/browse/LUCENE-8717) by Alan Woodward (@romseygeek), updated Apr 01 2019
Attachments: [LUCENE-8717.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8717/LUCENE-8717.patch) (versions: 2)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.