apache / apache/arrow-rs

[Parquet] Expose parquet split block bloom filter(SBBF) implementation for downstream libraries

Open
#8,277 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Hi, I come from the Lance community and is working on a new [Bloom Filter feature](https://github.com/lancedb/lance/issues/4517) for the project. TLDR Lance is a modern columnar data format designed for AI/ML workflow and is closely couple with the Arrow eco-system. I would like to directly use the SBBF from parquet rust implementation.

**Describe the solution you'd like**

Make the parquet bloom filter public accessible rather than [crate only](https://github.com/apache/arrow-rs/blob/main/parquet/src/bloom_filter/mod.rs#L253).

**Describe alternatives you've considered**

Creating a folk of the bloom filter in Lance is doable, but much less than ideal. There are other some [SBBF crates](https://crates.io/crates/sbbf-rs-safe/0.2.1/dependencies) out there but with very limited usage.

**Additional context**

cc @westonpace

Contributor guide

Open the contributing guide

Research direction

Start at parquet/src/bloom_filter/mod.rs, especially the crate-only implementation around line 253 referenced in the issue. Check how the parquet bloom filter is exposed and verify that downstream Rust libraries can access the SBBF implementation through the public API.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.