apache / apache/lucene

Lucene's facets should tap into `IndexSearcher`'s `TaskExecutor` too?

Open
#13,503 2 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Spinoff from the exciting discussion on https://github.com/apache/lucene/pull/13472:

Lucene has made great gains recently on intra-query concurrency: using multiple threads (with a "slice" work unit = one or more segments) to reduce latency of queries. Besides faster wall clock time, since CPU is running concurrently, Lucene also can gain efficiency because segments can terminate earlier / start using skipping as the more competitive results from other segments arrive sooner/concurrently, causing less total CPU to be spent to get the top hits for the query.

But I think Lucene's doc values and taxonomy facets do not use any concurrency? Even if you pass a `TaskExecutor` to `IndexSearcher`, facet counting will still run single threaded. Can we fix this to also make facet counting faster (net elapsed wall clock time)? It's tricky because some facet counting aggregate into data structures (like `int[]` or an HPPC int->int map) that are not easily made thread safe?

Note: we do have `ConcurrentSortedSetDocValuesFacetCounts` which does use concurrency, but the other facet counting (numeric ranges, taxonomy facets) do not. Also, `ConcurrentSortedSetDocValuesFacetCounts` takes its own `ExecutorService` not a `TaskExecutor`.

Contributor guide

Open the contributing guide

Research direction

Start by reading IndexSearcher’s TaskExecutor integration and the facet counting implementations mentioned in the issue, including numeric ranges, taxonomy facets, and ConcurrentSortedSetDocValuesFacetCounts. Determine how facet aggregation could use the shared executor without unsafe updates; done means the relevant facet counters use that concurrency path while preserving correct counts and improving elapsed query time.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.