apache / apache/lucene

A Time-limiting collector that works with CollectorManagers [LUCENE-8319]

Open
#9,366 5 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Currently Lucene has **TimeLimitingCollector** to support time-bound collection and it will throw 
**TimeExceededException** if timeout happens. This only works nicely with the single-thread low-level API from the IndexSearcher. The method signature is –

**void search(List<LeafReaderContext> leaves, Weight weight, Collector collector)**

The intended use is to always enclose the searcher.search(query, collector) call with a try ... catch and handle the timeout exception. Unfortunately when working with a **CollectorManager** in the multi-thread search context, the **TimeExceededException** thrown during collecting one leaf slice will be re-thrown by **IndexSearcher** without calling **CollectorManager**'s reduce(), even if other slices are successfully collected. The signature
of the search api with **CollectorManager** is –

**<C extends Collector, T> T search(Query query, CollectorManager<C, T> collectorManager)**
 
The good news is that IndexSearcher handles **CollectionTerminatedException** gracefully by ignoring it. We can either wrap TimeLimitingCollector and throw **CollectionTerminatedException** when timeout happens or simply replace **TimeExceededException** with **CollectionTerminatedException**. In either way, we also need to maintain a flag that indicates if timeout occurred so that the user know it's a partial collection.

---
Migrated from [LUCENE-8319](https://issues.apache.org/jira/browse/LUCENE-8319) by Tony Xu (@Tony-X), updated Sep 02 2020

Contributor guide

Open the contributing guide

Research direction

Start by reading TimeLimitingCollector, TimeExceededException, CollectionTerminatedException, and the IndexSearcher search API that accepts a CollectorManager. Trace how timeout exceptions from one leaf slice affect CollectorManager.reduce(). Done means timed-out collection is treated as partial collection, reduce() can still run for successful slices, and callers can detect that a timeout occurred.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.