apache / apache/lucene

PassageScorer could take proximity of terms into account [LUCENE-10011]

Open
#11,050 1 comment 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The UnifiedHighlighter scores its highlighted passages using a modified term frequency calculation, similar to BM25.  This means that two passages containing the same set of terms will score equivalently.  Given that proximity is often a reasonable proxy for relevance, and that passages contain the offsets of their internal hits, it would be useful to add the option of also weighting by the difference between the start of the first hit and the end of the last within the passage.

---
Migrated from [LUCENE-10011](https://issues.apache.org/jira/browse/LUCENE-10011) by Alan Woodward (@romseygeek)

Contributor guide

Open the contributing guide

Research direction

Start at the UnifiedHighlighter's PassageScorer and inspect how passage hit offsets are exposed. Determine how an optional proximity weighting based on the first hit's start and last hit's end should interact with the existing scoring. Done means passages with different term spans can receive different scores when the option is enabled, while the existing behavior remains available.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.