apache / apache/lucene

Remove FieldMaskingSpanQuery (or fix its scoring) [LUCENE-3723]

Open
#4,797 2 comments 0 reactions 0 assignees View on GitHub
affects-version:4.0-ALPHA legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In Lucene 4.0 we added new scoring mechanisms, but FieldMaskingSpanQuery is a serious problem:

Because it lies about the fields of its terms, this sometimes results in totally bogus
statistics, cases where a single terms totalTermFreq exceeds sumTotalTermFreq for the entire field (since its lying about it).

Such lying could result in NaN/Inf/Negative scores, exceptions, divide by zero, and other problems,
because the statistics are impossibly bogus.

---
Migrated from [LUCENE-3723](https://issues.apache.org/jira/browse/LUCENE-3723) by Robert Muir (@rmuir)
Attachments: [LUCENE-3723_test.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3723/LUCENE-3723_test.patch), [LUCENE-3723.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3723/LUCENE-3723.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reading the FieldMaskingSpanQuery implementation and the attached LUCENE-3723_test.patch. Investigate how its field reporting produces inconsistent term statistics and can lead to invalid scores; done means either removing the query or correcting its scoring so these impossible statistics and resulting failures no longer occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.