apache / apache/lucene

Early Terminating CollectorManager [LUCENE-8929]

Open
#9,972 15 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

We should have an early terminating collector manager which accurately tracks hits across all of its collectors and determines when there are enough hits, allowing all the collectors to abort.

The options for the same are:

1) Shared total count : Global "scoreboard" where all collectors update their current hit count. At the end of each document's collection, collector checks if N > threshold, and aborts if true

2) State Reporting Collectors: Collectors report their total number of counts collected periodically using a callback mechanism, and get a proceed or abort decision.

1) has the overhead of synchronization in the hot path, 2) can collect unnecessary hits before aborting.

I am planning to work on 2), unless objections

---
Migrated from [LUCENE-8929](https://issues.apache.org/jira/browse/LUCENE-8929) by Atri Sharma (@atris), updated Dec 30 2020
Parent: #9983

Contributor guide

Open the contributing guide

Research direction

No source file or test is named; begin by locating the existing CollectorManager and reviewing parent #9983 and the linked design discussion. Confirm the callback-based state-reporting approach, then validate that all collectors stop once the aggregate hit threshold is reached.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.