apache / apache/lucene

Always collect sparsely in TaxonomyFacets & switch to dense if there are enough unique labels

Open
#12,576 6 comments 0 reactions 0 assignees View on GitHub
module:facet type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Right now (at least int [`IntTaxonomyFacets`](https://github.com/apache/lucene/blob/main/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/IntTaxonomyFacets.java#L75)), we choose an integer array for counting if the number of hits is greater than 10% of the index; else we count sparsely. Counting densely may still not be a good idea if we have a huge number of hits but they only belong to a few labels - in effect making the values sparse.

We can use something like [DocIdSetBuilder](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/DocIdSetBuilder.java.) does: At first it uses a sparse structure to gather documents, and then upgrades to a non-sparse bit set once enough hits match.

[`FloatTaxonomyFacets` seems to always collect densely](https://github.com/apache/lucene/blob/main/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/FloatTaxonomyFacets.java#L65). Maybe it's also worth adding this decision making to the parent class instead?

Contributor guide

Open the contributing guide

Research direction

Start by reading IntTaxonomyFacets.java and FloatTaxonomyFacets.java, then compare their collection choices with DocIdSetBuilder.java. Determine how sparse collection should upgrade to dense when enough unique labels are matched, and verify that the decision applies consistently across the relevant TaxonomyFacets implementations.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.