apache / apache/lucene

Introduce Bloom Filter as non-experimental/core postings format

Open
#12,986 6 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Today, [BloomFilteringPostingsFormat](https://github.com/apache/lucene/blob/main/lucene/codecs/src/java/org/apache/lucene/codecs/bloom/BloomFilteringPostingsFormat.java) in Lucene is marked experimental.

Based on our analysis of the the data structure in OpenSearch for the Primary Key field using the `nyc_taxis` workload [[See Issue](https://github.com/opensearch-project/OpenSearch/issues/4489#issuecomment-1724998489)], we have found that it proves very useful for indexing performance, and also certain term queries/Get Document calls on the PK.

We want to introduce this as an opt-in feature in OpenSearch for customers so that they can take advantage of the performance improvements, and wanted inputs from the community on the following:

1. Why is the BloomFilteringPostingsFormat in experimental status?
2. Is it possible to contribute and mark it as a core feature with support for backward compatibility in Lucene?

We've done few changes in OpenSearch to support an off-heap implementation and introduce certain knobs which may prove useful for Lucene customers and would like to see if they can be contributed to Lucene as well [opensearch-project/OpenSearch/pull/11027].

Contributor guide

Open the contributing guide

Research direction

Start with lucene/codecs/src/java/org/apache/lucene/codecs/bloom/BloomFilteringPostingsFormat.java and review the linked OpenSearch analysis and pull request 11027. Investigate why the format is experimental and whether an opt-in, off-heap implementation with configurable knobs and backward compatibility is appropriate for Lucene. Done requires a defined community-supported core-feature path, not just a local code change.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.