apache / apache/lucene

DrillSideways and early termination

Open
#15,220 1 comment 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

As I was upgrading an application using Lucene 8.10 to Lucene 10.x, I've hit an unexpected situation with DrillSideways. In the past, when using DrillSideways with a collector that would early terminate, the whole collection would terminate and facet collection would also exit (with incomplete counts). It seems like now, DrillSideways#search code will wrap the hitCollector with a MultiCollector, and because of this, early termination in one collector doesn't translate into early termination in the other (the FacetCollector), so the collection will complete to the end. While I haven't test it, I expect FacetCollectorManager#search to behave similarly, since it's also wrapping the hitCollector in a MultiCollector.

Looking at the 8.x code, it seems like the behavior wasn't 100% consistent (i.e [when "drillDownDims" was empty](https://github.com/apache/lucene-solr/blob/faca81bff1a87ecbc6bc115d0cd79895fe12dd7e/lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java#L214) we would the and likely hit the same behavior) so this could have been a feature or a bug in 8.10 and before.

That said, I think this is a valid feature. I'm thinking we could have DrillSideways/FacetCollectorManager use a different implementation equivalent to MultiCollector that optionally exits collection whenever it sees the hitCollector throw a `CollectionTerminatedException`. Alternatively, if this is useful for features other than facet, we could change MultiCollector to optionally bubble up the `CollectionTerminatedException` instead of removing the particular collector that wants to terminate.

### Version and environment details

_No response_

Contributor guide

Open the contributing guide

Research direction

Read DrillSideways#search and FacetCollectorManager#search, then compare their MultiCollector usage with the Lucene 8.x behavior described in the issue. Reproduce the interaction with an early-terminating hit collector and determine the expected CollectionTerminatedException behavior; done means the agreed behavior is covered for both paths without regressing facet counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.