Always collect sparsely in TaxonomyFacets & switch to dense if there are enough unique labels
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Right now (at least int [`IntTaxonomyFacets`](https://github.com/apache/lucene/blob/main/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/IntTaxonomyFacets.java#L75)), we choose an integer array for counting if the number of hits is greater than 10% of the index; else we count sparsely. Counting densely may still not be a good idea if we have a huge number of hits but they only belong to a few labels - in effect making the values sparse.
We can use something like [DocIdSetBuilder](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/DocIdSetBuilder.java.) does: At first it uses a sparse structure to gather documents, and then upgrades to a non-sparse bit set once enough hits match.
[`FloatTaxonomyFacets` seems to always collect densely](https://github.com/apache/lucene/blob/main/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/FloatTaxonomyFacets.java#L65). Maybe it's also worth adding this decision making to the parent class instead?
Contributor guide
Research direction
Start by reading IntTaxonomyFacets.java and FloatTaxonomyFacets.java, then compare their collection choices with DocIdSetBuilder.java. Determine how sparse collection should upgrade to dense when enough unique labels are matched, and verify that the decision applies consistently across the relevant TaxonomyFacets implementations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100