apache / apache/lucene

SimpleSpanFragmenter can create very long fragments [LUCENE-5019]

Open
#6,083 1 comment 0 reactions 0 assignees View on GitHub
affects-version:4.3 legacy-jira-priority:Minor module:highlighter type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In SimpleSpanFragmenter, when a query term is followed by a stop word, the fragment will run until the end of the document.

When a query term is encountered (line 80), SimpleSpanFragmenter waits for the token following it before allowing the fragment to end (lines 68 to 72). When a stop word follows the query word (or any token with a position increment greater than 1), its position is skipped and the token SimpleSpanFragmenter is waiting for never arrive.

The attached patch fixes that by waiting for the first token following the query word instead of the token at the position after the query term.

---
Migrated from [LUCENE-5019](https://issues.apache.org/jira/browse/LUCENE-5019) by Alexandre Patry (@apatry)
Attachments: [simple-span-fragmenter.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5019/simple-span-fragmenter.patch)

Contributor guide

Open the contributing guide

Research direction

Locate SimpleSpanFragmenter and inspect the behavior described around lines 68-80. Reproduce a query term followed by a stop word or a token with a position increment greater than 1, then verify that the resulting fragment ends normally instead of running to the end of the document.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.