[Parquet] Expose parquet split block bloom filter(SBBF) implementation for downstream libraries
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Hi, I come from the Lance community and is working on a new [Bloom Filter feature](https://github.com/lancedb/lance/issues/4517) for the project. TLDR Lance is a modern columnar data format designed for AI/ML workflow and is closely couple with the Arrow eco-system. I would like to directly use the SBBF from parquet rust implementation.
**Describe the solution you'd like**
Make the parquet bloom filter public accessible rather than [crate only](https://github.com/apache/arrow-rs/blob/main/parquet/src/bloom_filter/mod.rs#L253).
**Describe alternatives you've considered**
Creating a folk of the bloom filter in Lance is doable, but much less than ideal. There are other some [SBBF crates](https://crates.io/crates/sbbf-rs-safe/0.2.1/dependencies) out there but with very limited usage.
**Additional context**
cc @westonpace
Contributor guide
Research direction
Start at parquet/src/bloom_filter/mod.rs, especially the crate-only implementation around line 253 referenced in the issue. Check how the parquet bloom filter is exposed and verify that downstream Rust libraries can access the SBBF implementation through the public API.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100