apache / apache/lucene

Allow extensions of IndexSearcher to provide custom SliceExecutor and slices computation

Open
#12,347 15 comments 0 reactions 1 assignee Claimed by @javanna View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

For concurrent segment search, lucene uses the slices method to compute the number of work units which can be processed concurrently.

a) It calculates slices in the [constructor of IndexSearcher](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java#L239) with default thresholds for document count and segment counts.
b) Provides an implementation of [SliceExecutor (i.e. QueueSizeBasedExecutor)](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java#L1008) based on executor type which applies the backpressure in concurrent execution based on a limiting factor of 1.5 times the passed in threadpool maxPoolSize.

In OpenSearch, there is a search threadpool which serves the search request to all the lucene indices (or OpenSearch shards) assigned to a node. Each node can get the requests to some or all the indices on that node.
I am exploring a mechanism such that I can dynamically control the max slices for each lucene index search request. For example: search requests to some indices on that node to have max 4 slices each and others to have 2 slices each. Then the threadpool shared to execute these slices does not have any limiting factor. In this model the top level search threadpool will limit the number of active search requests which will limit the number of work units in the SliceExecutor threadpool.

For this the derived implementation of IndexSearcher can get an input value in the constructor to control the slice count computation. Even though the `slice` method is `protected` it gets called from the constructor of base `IndexSearcher` class which prevents the derived class from using the passed in input.

To achieve this I am making change along the lines as suggested on discussion thread in dev mailing list to get some feedback

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.