Stop duplicating per-segment work across segment partitions
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
With the introduction of intra-segment concurrency support (see #13542), the `IndexSearcher` is able to parallelize search across partitions of segments. Some of our queries, like `PointRangeQuery`, perform per-segment computation ahead of time, that is currently not shared across segment partitions, but rather duplicated across them.
This makes intra-segment concurrency not appealing for those queries, which we should look at addressing. If we can share such computation across segment partitions, we can likely enable intra-segment slicing by default and ensure that intra-segment concurrency is beneficial for all queries.
Contributor guide
Research direction
Start by reading IndexSearcher and PointRangeQuery, then review issue #13542 for the intra-segment partitioning design. Trace where per-segment computation is performed for partitioned searches and determine how it could be shared; done means the work is no longer duplicated across partitions and intra-segment concurrency is beneficial for these queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100