apache / apache/lucene

Regarding the frequency used for scoring sloppy phrase queries.

Open
#13,152 2 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

I've been using Lucene (through OpenSearch) for querying and scoring human-written documents (!= logs). I often use sloppy phrase queries to handle languages variations which express similar meaning.

For example, one of my query contained `"frauduleus faillissement"~3` (in Dutch. This is equivalent to "fraudulent bankruptcy" in English). Thanks to the sloppy match, the following two sentences are correctly matched by that query:
1. Dit faillissement is frauduleus (This bankruptcy is fraudulent)
2. Dit is een frauduleus faillissement (This is a fraudulent bankruptcy)

While the two sentences are very similar in meaning, the way they are scored by the `SloppyPhraseScorer` is very different. According to the following lines the sentences will have respectively a frequency of `0.25` and `1`, leading to a large difference in the resulting relevance score.
https://github.com/apache/lucene/blob/3ce9ba9fd51a9b4e7228d81e19acbdb8b18f4e12/lucene/core/src/java/org/apache/lucene/search/SloppyPhraseMatcher.java#L166-L169,

While I understand the intent of `sloppyWeight()` to penalize sloppy matches that are different from the exact match, I feel that the penalty is way too strong. As a user, I deliberately make the choice to look for sloppy phrase matches and I wouldn't expect that such a strong penalty would apply. For that particular example, I would obtain a more accurate scoring by using multiple exact phrase queries.

Browsing the history of `SloppyPhraseScorer`, I see that this scoring approach has been in place for years and I did not find any issue discussing that implementation. Even though my use case might be niche, I believe a revision of that scoring method could greatly benefit applications on human-written texts.

Contributor guide

Open the contributing guide

Research direction

Start with the linked SloppyPhraseMatcher.java lines around sloppyWeight(), then read the surrounding SloppyPhraseScorer implementation and its history. Reproduce the two Dutch sentence examples and compare their frequencies; done would require an agreed revision to sloppy phrase scoring, which the issue does not yet specify.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.