apache / apache/lucene

it's not possible to access sub-query's freq information if BooleanScorer is use [LUCENE-2684]

Open
#3,758 14 comments 0 reactions 0 assignees View on GitHub
legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-priority:Major module:core/search type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

#3664 added an advanced feature, allowing an app to gather all sub-scorers for any Query.

This is powerful because then, during collection, the app can get some details about how each sub-query "participated" in the overall match for the given document.

However, I think this is completely broken if the BooleanQuery uses BooleanScorer, because that scorer is not doc-at-once. Instead, it batch processes chunks of 2048 sequential docIDs per scorer. This is a big performance gain, but it means that the sub scorers will all be positioned to the end of the 2048 doc chunk while the docs that matched within that chunk are collected.

I don't think we can easily fix this... likely the "fix" is to make it easy(ier) to force BQ to use BooleanScorer2 (which is doc-at-once)? It is actually possible to force this, today, by having your collector return false from acceptDocsOutOfOrder...

---
Migrated from [LUCENE-2684](https://issues.apache.org/jira/browse/LUCENE-2684) by Michael McCandless (@mikemccand), updated May 09 2016

Contributor guide

Open the contributing guide

Research direction

Start with the BooleanQuery path that uses BooleanScorer and compare it with the forced BooleanScorer2 behavior when acceptDocsOutOfOrder is false. Verify that sub-scorer frequency information reflects each collected document rather than the end of the processed 2048-document chunk.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.