Explore a single scoring implementation in DrillSidewaysScorer [LUCENE-10037]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
`DrillSidewaysScorer` currently implements three separate strategies for bulk scoring documents: `doQueryFirstScoring`, `doUnionScoring` and `doDrillDownAdvanceScoring`. As far as I can tell, this code dates back to 2013 and two of the three approaches appear to emulate the `BooleanScorer` "window scoring" / "term-at-a-time" strategy. While this strategy in `BooleanScorer` is still useful in some cases, the primary benefit, from what I can tell, is to avoid re-heap operations in disjunction cases (as recently [described](http://mail-archives.apache.org/mod_mbox/lucene-dev/202106.mbox/%3CCAPsWd%2BMbYckCR2LHxHy4-%3DoZPnvX%3D9Er8hwb%2BG76jHb85JePvw%40mail.gmail.com%3E) by @jpountz). I can't see any reason why we'd prefer these two approaches anymore in `DrillSidewaysScorer` since we're doing pure conjunctions (no re-heaping to worry about) and `doQueryFirstScoring` takes advantage of skipping by advancing postings (while the other two approaches iterate their postings entirely, only relying on nextDoc functionality). Finally, we added an optimization (#11069) that can only work for `doQueryFirstScoring` that lazily evaluates the `score` (where-as `doUnionScoring` and `doDrillDownAdvanceScoring` eagerly evaluate it).
All this is to say we should try sending all scoring through `doQueryFirstScoring` and see how it benchmarks. I'm not sure if we have benchmarks setup already for drill sideways, but I'd love to see if we can't optimize `DrillSidewaysScorer` while also reducing its code complexity!
---
Migrated from [LUCENE-10037](https://issues.apache.org/jira/browse/LUCENE-10037) by Greg Miller (@gsmiller), updated Jul 28 2021
Contributor guide
Research direction
Start with DrillSidewaysScorer and compare doQueryFirstScoring, doUnionScoring, and doDrillDownAdvanceScoring, including the existing drill-sideways benchmarks if available. Measure whether routing scoring through doQueryFirstScoring preserves behavior and improves performance; done means benchmark results support the change and the redundant strategies can be removed safely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100