Introduce Bloom Filter as non-experimental/core postings format
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Today, [BloomFilteringPostingsFormat](https://github.com/apache/lucene/blob/main/lucene/codecs/src/java/org/apache/lucene/codecs/bloom/BloomFilteringPostingsFormat.java) in Lucene is marked experimental.
Based on our analysis of the the data structure in OpenSearch for the Primary Key field using the `nyc_taxis` workload [[See Issue](https://github.com/opensearch-project/OpenSearch/issues/4489#issuecomment-1724998489)], we have found that it proves very useful for indexing performance, and also certain term queries/Get Document calls on the PK.
We want to introduce this as an opt-in feature in OpenSearch for customers so that they can take advantage of the performance improvements, and wanted inputs from the community on the following:
1. Why is the BloomFilteringPostingsFormat in experimental status?
2. Is it possible to contribute and mark it as a core feature with support for backward compatibility in Lucene?
We've done few changes in OpenSearch to support an off-heap implementation and introduce certain knobs which may prove useful for Lucene customers and would like to see if they can be contributed to Lucene as well [opensearch-project/OpenSearch/pull/11027].
Contributor guide
Research direction
Start with lucene/codecs/src/java/org/apache/lucene/codecs/bloom/BloomFilteringPostingsFormat.java and review the linked OpenSearch analysis and pull request 11027. Investigate why the format is experimental and whether an opt-in, off-heap implementation with configurable knobs and backward compatibility is appropriate for Lucene. Done requires a defined community-supported core-feature path, not just a local code change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100