apache / apache/lucene

DrillSideways does not support intra-segment concurrency

Open
#13,753 1 comment 0 reactions 0 assignees View on GitHub
discussion
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

With the introduction of intra-segment search concurrency (see #13542), `DrillSideways` is the only case that disables it in tests which can't support intra-segment slicing given its implementation requirements to go through all docs in a segment at once. This is asserted as well in its score method which checks that min and max are respectively `0` and `NO_MORE_DOCS`.

With intra-segment slicing randomly enabled in tests, we need to explicitly disable intra-segment concurrency in any test that relies on `DrillSideways`. That required adding an additional `newSearcher` method that takes in the supported concurrency: none, inter-segment or intra-segment.

This issue is to discuss a plan to support intra-segment concurrency in `DrillSideways`. Ideally, we'd be able to one day enable intra-segment slicing by default, once #13745 is addressed, but that would not make `IndexSearcher` usable in `DrillSideways` out of the box which is not a good user experience.

Contributor guide

Open the contributing guide

Research direction

Start by reading the DrillSideways implementation and its score method, then review #13542 for intra-segment search concurrency and #13745 for the related limitation. Define a concrete design for supporting intra-segment slicing without requiring callers to disable concurrency, and document the expected behavior and tests needed to verify it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.