SynonymGraphFilter cannot handle input stream with tokens filtered. [LUCENE-8985]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
@janhoy find the bug.
In an analyzer with e.g. stopFilter where tokens are removed from the stream and replaced with a “hole”, synonymgraphfilter will not preserve these holes but remove them, resulting in certain phrase queries failing.
---
Migrated from [LUCENE-8985](https://issues.apache.org/jira/browse/LUCENE-8985) by Chongchen Chen, updated Sep 02 2020
Attachments: [SGF_SF_interaction.patch.txt](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8985/SGF_SF_interaction.patch.txt)
Contributor guide
Research direction
Start with the attached SGF_SF_interaction.patch.txt and reproduce an analyzer combining a stop filter with SynonymGraphFilter. Inspect how removed tokens and their holes are represented, then verify that phrase queries retain the expected token positions when the fix is complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100