SimpleSpanFragmenter can create very long fragments [LUCENE-5019]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
In SimpleSpanFragmenter, when a query term is followed by a stop word, the fragment will run until the end of the document.
When a query term is encountered (line 80), SimpleSpanFragmenter waits for the token following it before allowing the fragment to end (lines 68 to 72). When a stop word follows the query word (or any token with a position increment greater than 1), its position is skipped and the token SimpleSpanFragmenter is waiting for never arrive.
The attached patch fixes that by waiting for the first token following the query word instead of the token at the position after the query term.
---
Migrated from [LUCENE-5019](https://issues.apache.org/jira/browse/LUCENE-5019) by Alexandre Patry (@apatry)
Attachments: [simple-span-fragmenter.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5019/simple-span-fragmenter.patch)
Contributor guide
Research direction
Locate SimpleSpanFragmenter and inspect the behavior described around lines 68-80. Reproduce a query term followed by a stop word or a token with a position increment greater than 1, then verify that the resulting fragment ends normally instead of running to the end of the document.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100