apache / apache/lucene

Explore a single scoring implementation in DrillSidewaysScorer [LUCENE-10037]

Open
#11,076 2 comments 0 reactions 0 assignees View on GitHub
affects-version:9.0 legacy-jira-priority:Minor module:facet type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

`DrillSidewaysScorer` currently implements three separate strategies for bulk scoring documents: `doQueryFirstScoring`, `doUnionScoring` and `doDrillDownAdvanceScoring`. As far as I can tell, this code dates back to 2013 and two of the three approaches appear to emulate the `BooleanScorer` "window scoring" / "term-at-a-time" strategy. While this strategy in `BooleanScorer` is still useful in some cases, the primary benefit, from what I can tell, is to avoid re-heap operations in disjunction cases (as recently [described](http://mail-archives.apache.org/mod_mbox/lucene-dev/202106.mbox/%3CCAPsWd%2BMbYckCR2LHxHy4-%3DoZPnvX%3D9Er8hwb%2BG76jHb85JePvw%40mail.gmail.com%3E) by @jpountz). I can't see any reason why we'd prefer these two approaches anymore in `DrillSidewaysScorer` since we're doing pure conjunctions (no re-heaping to worry about) and `doQueryFirstScoring` takes advantage of skipping by advancing postings (while the other two approaches iterate their postings entirely, only relying on nextDoc functionality). Finally, we added an optimization (#11069) that can only work for `doQueryFirstScoring` that lazily evaluates the `score` (where-as `doUnionScoring` and `doDrillDownAdvanceScoring` eagerly evaluate it).

 

All this is to say we should try sending all scoring through `doQueryFirstScoring` and see how it benchmarks. I'm not sure if we have benchmarks setup already for drill sideways, but I'd love to see if we can't optimize `DrillSidewaysScorer` while also reducing its code complexity!

---
Migrated from [LUCENE-10037](https://issues.apache.org/jira/browse/LUCENE-10037) by Greg Miller (@gsmiller), updated Jul 28 2021

Contributor guide

Open the contributing guide

Research direction

Start with DrillSidewaysScorer and compare doQueryFirstScoring, doUnionScoring, and doDrillDownAdvanceScoring, including the existing drill-sideways benchmarks if available. Measure whether routing scoring through doQueryFirstScoring preserves behavior and improves performance; done means benchmark results support the change and the redundant strategies can be removed safely.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.