apache / apache/lucene

UnInverted cache uses term freq to filter out terms (but deleted docs are included in the freq count) [LUCENE-4242]

Open
#5,312 12 comments 0 reactions 0 assignees View on GitHub
affects-version:4.0-BETA legacy-jira-priority:Minor module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

TermEnum.docFreq() count is used to compute uninverted index
(DocTermOrds.uninvert()). The code goes like:

final int df = te.docFreq();
if (df <= maxTermDocFreq) {

So, if there are deleted documents in the index and maxTermDocFreq is
low, then the term will be excluded (even if the freq of the livedocs
is OK). Most likely, the cache will be incomplete.

---
Migrated from [LUCENE-4242](https://issues.apache.org/jira/browse/LUCENE-4242) by Roman Chyla, updated Jul 24 2012
Attachments: [LUCENE-4242.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4242/LUCENE-4242.patch) (versions: 3)

Contributor guide

Open the contributing guide

Research direction

Start in DocTermOrds.uninvert() at the TermEnum.docFreq() and maxTermDocFreq filtering described in the issue, then review the attached LUCENE-4242.patch. Verify behavior with deleted documents and confirm that terms are not incorrectly excluded when the live-document frequency is within the limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.