apache / apache/arrow-rs

Expose public async bloom filter reader (metadata + `AsyncFileReader`)

Open
#9,067 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

I'm integrating Parquet bloom filters into an async pruning pipeline and found a gap in the public API.

For current situation,

- There is a sync API:
`Sbbf::read_from_column_chunk(column_meta, reader)`
- There is an async method, but only on the async Arrow builder:
`ParquetRecordBatchStreamBuilder::get_row_group_column_bloom_filter(...)`
- The helper used internally to parse bloom filter headers is `pub(crate)`:
`chunk_read_bloom_filter_header_and_offset` (in `parquet::bloom_filter`)

If parquet crate only has `ParquetMetaData` + an `AsyncFileReader`, downstream applications can't read bloom filters without re‑implementing Parquet bloom header parsing.

This blocks async metadata‑only pruning libraries (like me) from using bloom filters safely and efficiently.

**Describe the solution you'd like**

Expose a public async bloom reader that mirrors the sync API:

```rust
pub async fn read_bloom_filter_async(
column_meta: &ColumnChunkMetaData,
reader: &mut R
) -> Result>;
```

This would:

- keep internal header parsing private
- allow async pruning without coupling to Arrow builder
- avoid duplicate parsing logic in downstream crates
- be backwards compatible (pure API addition)

*Alternative*
Make `chunk_read_bloom_filter_header_and_offset` public, but this is a low‑level parsing helper and would bake in more implementation detail.

Contributor guide

Open the contributing guide

Research direction

Start in parquet::bloom_filter by reading chunk_read_bloom_filter_header_and_offset, then compare Sbbf::read_from_column_chunk with ParquetRecordBatchStreamBuilder::get_row_group_column_bloom_filter and the AsyncFileReader interface. Done means a public async reader accepts ColumnChunkMetaData and an AsyncFileReader, returns the requested bloom filter result, and keeps internal header parsing private.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.