A Time-limiting collector that works with CollectorManagers [LUCENE-8319]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Currently Lucene has **TimeLimitingCollector** to support time-bound collection and it will throw
**TimeExceededException** if timeout happens. This only works nicely with the single-thread low-level API from the IndexSearcher. The method signature is –
**void search(List<LeafReaderContext> leaves, Weight weight, Collector collector)**
The intended use is to always enclose the searcher.search(query, collector) call with a try ... catch and handle the timeout exception. Unfortunately when working with a **CollectorManager** in the multi-thread search context, the **TimeExceededException** thrown during collecting one leaf slice will be re-thrown by **IndexSearcher** without calling **CollectorManager**'s reduce(), even if other slices are successfully collected. The signature
of the search api with **CollectorManager** is –
**<C extends Collector, T> T search(Query query, CollectorManager<C, T> collectorManager)**
The good news is that IndexSearcher handles **CollectionTerminatedException** gracefully by ignoring it. We can either wrap TimeLimitingCollector and throw **CollectionTerminatedException** when timeout happens or simply replace **TimeExceededException** with **CollectionTerminatedException**. In either way, we also need to maintain a flag that indicates if timeout occurred so that the user know it's a partial collection.
---
Migrated from [LUCENE-8319](https://issues.apache.org/jira/browse/LUCENE-8319) by Tony Xu (@Tony-X), updated Sep 02 2020
Contributor guide
Research direction
Start by reading TimeLimitingCollector, TimeExceededException, CollectionTerminatedException, and the IndexSearcher search API that accepts a CollectorManager. Trace how timeout exceptions from one leaf slice affect CollectorManager.reduce(). Done means timed-out collection is treated as partial collection, reduce() can still run for successful slices, and callers can detect that a timeout occurred.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100