Remove FieldMaskingSpanQuery (or fix its scoring) [LUCENE-3723]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
In Lucene 4.0 we added new scoring mechanisms, but FieldMaskingSpanQuery is a serious problem:
Because it lies about the fields of its terms, this sometimes results in totally bogus
statistics, cases where a single terms totalTermFreq exceeds sumTotalTermFreq for the entire field (since its lying about it).
Such lying could result in NaN/Inf/Negative scores, exceptions, divide by zero, and other problems,
because the statistics are impossibly bogus.
---
Migrated from [LUCENE-3723](https://issues.apache.org/jira/browse/LUCENE-3723) by Robert Muir (@rmuir)
Attachments: [LUCENE-3723_test.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3723/LUCENE-3723_test.patch), [LUCENE-3723.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3723/LUCENE-3723.patch)
Contributor guide
Research direction
Start by reading the FieldMaskingSpanQuery implementation and the attached LUCENE-3723_test.patch. Investigate how its field reporting produces inconsistent term statistics and can lead to invalid scores; done means either removing the query or correcting its scoring so these impossible statistics and resulting failures no longer occur.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100