Lucene's facets should tap into `IndexSearcher`'s `TaskExecutor` too?
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Spinoff from the exciting discussion on https://github.com/apache/lucene/pull/13472:
Lucene has made great gains recently on intra-query concurrency: using multiple threads (with a "slice" work unit = one or more segments) to reduce latency of queries. Besides faster wall clock time, since CPU is running concurrently, Lucene also can gain efficiency because segments can terminate earlier / start using skipping as the more competitive results from other segments arrive sooner/concurrently, causing less total CPU to be spent to get the top hits for the query.
But I think Lucene's doc values and taxonomy facets do not use any concurrency? Even if you pass a `TaskExecutor` to `IndexSearcher`, facet counting will still run single threaded. Can we fix this to also make facet counting faster (net elapsed wall clock time)? It's tricky because some facet counting aggregate into data structures (like `int[]` or an HPPC int->int map) that are not easily made thread safe?
Note: we do have `ConcurrentSortedSetDocValuesFacetCounts` which does use concurrency, but the other facet counting (numeric ranges, taxonomy facets) do not. Also, `ConcurrentSortedSetDocValuesFacetCounts` takes its own `ExecutorService` not a `TaskExecutor`.
Contributor guide
Research direction
Start by reading IndexSearcher’s TaskExecutor integration and the facet counting implementations mentioned in the issue, including numeric ranges, taxonomy facets, and ConcurrentSortedSetDocValuesFacetCounts. Determine how facet aggregation could use the shared executor without unsafe updates; done means the relevant facet counters use that concurrency path while preserving correct counts and improving elapsed query time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100