Early Terminating CollectorManager [LUCENE-8929]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
We should have an early terminating collector manager which accurately tracks hits across all of its collectors and determines when there are enough hits, allowing all the collectors to abort.
The options for the same are:
1) Shared total count : Global "scoreboard" where all collectors update their current hit count. At the end of each document's collection, collector checks if N > threshold, and aborts if true
2) State Reporting Collectors: Collectors report their total number of counts collected periodically using a callback mechanism, and get a proceed or abort decision.
1) has the overhead of synchronization in the hot path, 2) can collect unnecessary hits before aborting.
I am planning to work on 2), unless objections
---
Migrated from [LUCENE-8929](https://issues.apache.org/jira/browse/LUCENE-8929) by Atri Sharma (@atris), updated Dec 30 2020
Parent: #9983
Contributor guide
Research direction
No source file or test is named; begin by locating the existing CollectorManager and reviewing parent #9983 and the linked design discussion. Confirm the callback-based state-reporting approach, then validate that all collectors stop once the aggregate hit threshold is reached.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100