apache / apache/lucene

Stop duplicating per-segment work across segment partitions

Open
#13,745 13 comments 3 reactions 0 assignees View on GitHub
good-mentored-project type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

With the introduction of intra-segment concurrency support (see #13542), the `IndexSearcher` is able to parallelize search across partitions of segments. Some of our queries, like `PointRangeQuery`, perform per-segment computation ahead of time, that is currently not shared across segment partitions, but rather duplicated across them.

This makes intra-segment concurrency not appealing for those queries, which we should look at addressing. If we can share such computation across segment partitions, we can likely enable intra-segment slicing by default and ensure that intra-segment concurrency is beneficial for all queries.

Contributor guide

Open the contributing guide

Research direction

Start by reading IndexSearcher and PointRangeQuery, then review issue #13542 for the intra-segment partitioning design. Trace where per-segment computation is performed for partitioned searches and determine how it could be shared; done means the work is no longer duplicated across partitions and intra-segment concurrency is beneficial for these queries.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.