apache / apache/lucene

Wrong Formula in LMDirichletSimilarity [LUCENE-7480]

Open
#8,532 1 comment 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

It seems that LMDirichletSimilarity only calculates "score" method if the term occurs in the document. Otherwise, in line 389 of BooleanWeight (Lucene 6.2.0) subScorer becomes null, and thus the clause is not added to the optional list in order to be scored.

However, in the original formula of LM (http://www.stat.uchicago.edu/\~lafferty/pdf/smooth-tois.pdf, formula 6), we have "n log a_d" (n is the number of query terms). Therefore, even for the query terms not present in the document a "log a_d" must be added to the final score.

But the implementation of LMDirichletSimilarity adds "log a_d" to the score in the "score" method, and therefore it is only added to the final score for the query terms present in the document.

This can worsen the retrieval results compared to the correct formula. I tried to correct this for myself but because of the plenty of "final" methods and classes, I was not successful. Please, check the problem and solve it if approved, and also please tell me how I can correct it before a new release is published.

---
Migrated from [LUCENE-7480](https://issues.apache.org/jira/browse/LUCENE-7480) by Shayan Tabrizi, updated Oct 07 2016

Contributor guide

Open the contributing guide

Research direction

Start with LMDirichletSimilarity's score method and the subScorer handling at line 389 of BooleanWeight in Lucene 6.2.0. Compare the scoring flow with formula 6 in the linked paper, focusing on query terms absent from a document. Done means the final score accounts for those terms according to the formula and the retrieval behavior is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.